Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issues Cloning Spec repo - GitHub taking a very long time to download changes to the Specs Repo #4989

Closed
jlubeck opened this issue Mar 7, 2016 · 73 comments
Assignees
Labels
t3:discussion These are issues that can be non-issues, and encompass best practices, or plans for the future.

Comments

@jlubeck
Copy link

jlubeck commented Mar 7, 2016

Note from @orta -

If you are here because your Specs repo isn't updating, run: cd ~/.cocoapods/repos/master && git fetch --depth=2147483647 - this will convert your local repository of Podspecs to be a full clone, as opposed to a shallow copy.


What did you do?

Run pod setup

What did you expected to happen?

Clone Spec repo master

What happened instead?

It only downloads a few bytes and then throws error:

fatal: unable to access 'https://github.com/CocoaPods/Specs.git/': transfer closed with outstanding read data remaining

Podfile

No Podfile yet

I also tried cloning the repo manually or with the githhub desktop app with no avail.
I´m having no issues cloning any other repo in github. Only with this one. Is it possible there is something wrong with it???

Thanks

@jlubeck
Copy link
Author

jlubeck commented Mar 7, 2016

Tried again, new error:

error: RPC failed; result=18, HTTP code = 200
fatal: The remote end hung up unexpectedly
[!] /usr/bin/git clone https://github.com/CocoaPods/Specs.git master --depth=1

Cloning into 'master'...
error: RPC failed; result=18, HTTP code = 200
fatal: The remote end hung up unexpectedly

Very weird...

@ecch531
Copy link

ecch531 commented Mar 7, 2016

I got same issue, too.

@mangofever
Copy link

Same.
Cannot clone spec repo.

@art-divin
Copy link

+1, same issue
git clone https://github.com/CocoaPods/Specs.git takes forever

@stringsanbu
Copy link

+1, been messing around with this for awhile. I doubled the buffer, didn't work. Uninstalled and reinstalled pods, didn't work. Tried to clone manually, no cigar. It actually seems to be getting "something" but fails. Using verbose didn't say much, just said it had issues accessing it.

I tried accessing my other repos and it seemed to be OK, but it was definitely slower than normal.

@huinme
Copy link

huinme commented Mar 7, 2016

+1

I got same issue, too.
my pod version was "0.39.0"

I tried cloning master repos directly (by git clone git@github.com:CocoaPods/Specs.git master --depth=1 --verbose), but also failed.

@oronbz
Copy link

oronbz commented Mar 7, 2016

+1

2 similar comments
@aceontech
Copy link

+1

@pedrocid
Copy link

pedrocid commented Mar 7, 2016

+1

@MarkMolina
Copy link

+1. No success after increasing buffer / reinstall / manual clone

@stringsanbu
Copy link

Temporary workaround which might work: https://github.com/CocoaPods/Specs/archive/master.zip haven't tested though

wget https://github.com/CocoaPods/Specs/archive/master.zip

@samuel-mellert
Copy link

https://github.com/CocoaPods/Specs/archive/master.zip would be the correct link, I guess?

@fraxool
Copy link

fraxool commented Mar 7, 2016

Same error here. What should we do with the file at https://github.com/CocoaPods/Specs/archive/master.zip ?

@stringsanbu
Copy link

My bad, the wget link is correct. Just edited my first link.

@stringsanbu
Copy link

Not sure what to do with the file yet. Trying to see if we can manually run the commands to have pod setup work.

@aceontech
Copy link

Yeah, it merely downloads the repo's contents. The .git/ directory is missing, so it's not recognized as a git repo.

@samuel-mellert
Copy link

yes.. same here.. It always tries to clone the master repo. Even when I run it with --no-repo-update I get "Creating shallow clone of spec repo master-1 from https://github.com/CocoaPods/Specs.git"

@MarkMolina
Copy link

Did anyone try this with 1.0.0 beta 4?

@huinme
Copy link

huinme commented Mar 7, 2016

@MarkMolina I tried, but same result.

@czechboy0
Copy link
Contributor

Try to cd into ~/.cocoapods/repos/master, then git clean -fd to clean up the working copy, git checkout -- . to ensure you're on master and then git pull manually. This took ages but worked for me.

@SoundBlaster
Copy link

+1

@aceontech
Copy link

Thx, but I removed my master spec repo before I realized something was up with the Github repo ^.^

@stringsanbu
Copy link

Got a temp workaround! Tested with my app and everything is working. This is really only needed if you deleted the master repo. If the master folder is still in your ~/.cocoapods/repos folder with contents then you should be ok to just use pod install --no-repo-update.

  • Try doing a pod setup. This should at minimum download the .git to ~/.cocoapods/repos/master
  • While this is going, you need to move the .git folder from the master folder to somewhere temporary.
  • Stop the pod setup. Delete master folder in repos
  • Use the wget command below to get the zip of the repo
    wget https://github.com/CocoaPods/Specs/archive/master.zip
  • Unzip the master zip and move its contents to ~/.cocoapods/repos/master
  • Move your .git folder from wherever you put it to ~/.cocoapods/repos/master as .git
  • Go to your project folder, do a pod install --no-repo-update

And you should be good to go!

So in short, here is the basic list of commands I used:

pod setup (in a separate tab)
mv ~/.cocoapods/repos/master/.git ~/tempSpecsGitFolder
^C on pod setup tab
wget https://github.com/CocoaPods/Specs/archive/master.zip
open master.zip (unzipping it)
mv Specs-master ~/.cocoapods/repos/master
mv ~/tempSpecsGitFolder ~/.cocoapods/repos/master/.git
cd [project folder]
pod install --no-repo-update

@aceontech
Copy link

Is this a Cocoapods or a wider GitHub issue?

@stringsanbu
Copy link

@aceontech Pretty sure it is a GitHub issue, but my other repos are working fine so perhaps only certain repos on certain servers (on their backend) are affected.

@aceontech
Copy link

I was just able to do a successful pod setup. Don't know if it's repeatable.

AlexMacBookPro:repos alex$ pod setup --verbose

Setting up CocoaPods master repo

Creating shallow clone of spec repo `master` from `https://github.com/CocoaPods/Specs.git` (branch `master`)
  $ /usr/bin/git clone https://github.com/CocoaPods/Specs.git master --depth=1
  Cloning into 'master'...
  Checking out files: 100% (74393/74393), done.
  $ /usr/bin/git checkout master
  Already on 'master'
  Your branch is up-to-date with 'origin/master'.

@SoundBlaster
Copy link

Github is very very slow: ~ 40-50KB/s

@pedrocid
Copy link

pedrocid commented Mar 7, 2016

I was able to do pod setup -- verbose right now.

@segiddins
Copy link
Member

This is a GitHub issue rather than a cocoapods issue -- you're best off reporting it to their support rather than us, since there's nothing we can do about it.

@tychop
Copy link

tychop commented Mar 7, 2016

And github refers to Cocoapods... Great...

@solarce
Copy link

solarce commented Mar 8, 2016

Hey there, I am the infrastructure manager for Travis CI and my team is responsible for our OSX build images, which includes CocoaPods and the speed of the pod setup operation in each build is something we've been discussing lately.

We'd definitely love to hear more about we how could:

  1. Improve how we do an initial install and setup of cocoapods and the specs, when we build new images for new Xcode updates, e.g. 7.3b5 yesterday. Right now we just do https://github.com/travis-infrastructure/osx-image-bootstrap/blob/master/bootstrap.sh#L73-L75
  2. If there are some improvements we can make to the things we do at runtime, in https://github.com/travis-ci/travis-build/blob/master/lib/travis/build/script/objective_c.rb and/or how our build in caching handles the specs

@codeOfRobin
Copy link

Stupid Question. Are similar load problems seen in repos like Homebrew(which maintains similar kinds of information) as CocoaPods?

@ZevEisenberg
Copy link

I'd like to put in a plug for checking your Pods folder in to source control. I know there are arguments on both sides, but I've been checking in Pods for over a year on all projects and loving it. It removes your project's dependence on pod install in order to run on CI, which speeds up builds, and the repo size hit isn't really a big problem in my experience. One note: you'll generally want to change your Podfile and run pod install and merge that separately from any other changes, to keep your pull requests readable.

@jcampbell05
Copy link
Collaborator

@solarce Aside from using Travis CI caching (which is awesome BTW) and the ideas above (Which are a positive direction and great work on the quick response from @alloy and the team ). What I've been using as a solution for my company is a plugin I wrote which attempts to use the bundle install --deployment functionality.

Check it out here:
https://github.com/jcampbell05/cocoapods-deploy

It looks at the lockfile, and uses the raw HTTP links to the podspecs for each dependency and downloads them. This means that we don't need to clone the CocoaPods Specs repository at all, we can take advantage of Github's HTTP caching (Reducing the workload for @vmg and @mhagger) and TravisCI can cache these Podspecs as well (Since they reside in the Pods folder).

With this plugin we've experienced a 85% decrease in CocoaPods install times (Down from around 500 seconds to 85). I would love to know your thoughts on this and to see if it works for you.

@jcampbell05
Copy link
Collaborator

@ZevEisenberg I've never been a fan of that approach and always felt it could be something that should be solved via a CocoaPods's plugin (like the one above). Since other package managers solve this via a special deploy mode which doesn't require you to mix your dependencies and source together.

I think its important to give people the choice :)

@orta
Copy link
Member

orta commented Mar 8, 2016

OK, looks like it's time to lock this thread. Sorry people wanting to continue this conversation productively, but we've hit the hackernews/mainstream point where the tone of the conversation is going awry due to outsiders. If you'd like to reach out to me or the CocoaPods team, you can direct message me on twitter (same username) for a quick response, or use info@cocoapods.org for something team-y.

GitHub staff we totally appreciate the discussion, and will update this thread once we've got somewhere towards having a release with some fixes. I'll unlock it then 👍.

@CocoaPods CocoaPods locked and limited conversation to collaborators Mar 8, 2016
@alloy
Copy link
Member

alloy commented Mar 8, 2016

Shame :(

@segiddins
Copy link
Member

Just a quick update: I've just released CocoaPods v1.0.0.beta.6 with a few improvements to how we handle specs repos, particularly https://github.com/CocoaPods/Specs. The full CHANGELOG is available at https://github.com/CocoaPods/CocoaPods/releases/tag/1.0.0.beta.6, but in short:

  • Specs repos will no longer auto-update by default when running pod install
  • When attempting to pull the master specs repo, we check that preview API for a 304 and will not perform the fetch if the API says we have the latest commit
  • Shallow clones of specs repos are no longer supported
  • Shallow clones of the master specs repo will be automatically unshallowed.

In the next beta, we will be adding forward compatibility support for a sharded specs repo directory structure, to be migrated to in a few months.

We hope that this set of changes will lead to a better user experience in the long run for all users of CocoaPods, and will also help ease the burden of serving the master specs repository that GitHub faces. I speak on behalf of the team when I say thank you to everyone who has helped us over the past week, in particular @DanielTomlinson and @mrackwitz for making massive pull requests, @orta and @alloy for helping coordinate this effort, and the entire GitHub team, particularly @vmg, @mhagger, and @arthurnn for providing technical advice and infrastructure to CocoaPods, our users, and the entire open source community.

If anyone has further questions / comments about this release and the work to come, please feel free to reach out either to me personally (segiddins@segiddins.me) or to the CocoaPods team (info@cocoapods.org). Thank you everyone for your patience over the past week.

-- ❤️ The CocoaPods Team

@segiddins segiddins added the label Mar 15, 2016
@CocoaPods CocoaPods unlocked this conversation Mar 15, 2016
@kylef
Copy link
Contributor

kylef commented Mar 15, 2016

I'd suspect it might be a while for users to update to 1.0.0 considering it includes many breaking changes over previous versions. Does anyone think it would be worth back porting these fixes to the 0.39.x series and issuing a patch update removing shallow etc?

Especially if the master repository is going to be sharded and become incompatible with 0.39.

@jcampbell05
Copy link
Collaborator

@kylef I agree it would be good to have a 0.40 with some of these changes.

@vinnybad
Copy link

@kylef it would certainly help our teams!

@CocoaPods CocoaPods locked and limited conversation to collaborators Mar 15, 2016
@segiddins
Copy link
Member

Thanks everyone for sticking with us! We've written up a post-mortem of the issue and the steps taken to mitigate at http://blog.cocoapods.org/Master-Spec-Repo-Rate-Limiting-Post-Mortem/.

@dnkoutso
Copy link
Contributor

Please see https://blog.cocoapods.org/CocoaPods-1.7.2/ as version 1.7.2+ has switched to use CDN and fresh installs of CocoaPods do not need to clone the master specs repo anymore.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
t3:discussion These are issues that can be non-issues, and encompass best practices, or plans for the future.
Projects
None yet
Development

No branches or pull requests