Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download from Rackspace Doesn't always get all Segments #33

Open
homerlex opened this issue Oct 17, 2019 · 1 comment
Open

Download from Rackspace Doesn't always get all Segments #33

homerlex opened this issue Oct 17, 2019 · 1 comment

Comments

@homerlex
Copy link

We are using fog (fog-rackspace (0.1.6) to do offsite backups from Heroku to Rackspace. Our files are 6GB+ so we need to do the segmented uploads as described here: upload-large-files.

The upload works perfectly every time. When we manually download the file from the Rackspace UI it is fully intact.

The problem we are having is when we try downloading the file using fog we sometimes only get the first segment of the file. We can tell because the size of the file we see download is exactly 4GB (our UPLOAD_SEGMENT_LIMIT is set to 4GB). We are downloading the file with the suggested code from the above mentioned blog:

File.open('downloaded-file.jpg', 'w') do | f |
  directory.files.get("my_big_file.jpg") do | data, remaining, content_length |
    f.syswrite data
  end
end

The problem with the download is intermittent. Has anyone else experienced this issue?

@homerlex
Copy link
Author

I was just testing with an 11MB file with 3MB chunks and has the same issue. The problem is actuall that the last segment was not downloaded. I'm getting the feeling that perhaps we are trying to download the file too soon after updating it. Perhaps we need to sleep a bit in order for Rackspace to get its act together before we try the download? I'll experiment with this...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant