Some plugins aren't being mirrored #4
Comments
|
We just re-added this to the queue and it is currently about 10th in line. Please let me know if it is not updated within the next hour or so. |
|
Just a quick update. It is still in the queue but it is working its way to the top. |
|
Looks like the repo was created, but nothing got put in it. Think it could be because there's nothing in trunk because there's a There's a reason I prefer git when it comes to pulling plugins. This On Fri, Jan 30, 2015 at 2:17 PM, Mike Hansen [email protected]
|
|
http://www.pluginmirror.com/plugins/wordpress-importer also empty. We were just looking at this at The Loop stackexchange WP chatroom today. |
|
Exactly, when this reached the top of the queue and was mirrored there was no content in the trunk directory so nothing was copied over. |
|
See I came into this ass-backwards. My use case is basically an infrastructure that maintains a large amount of The only way to handle this in a sane fashion is some value of I came to discover, without knowing about the pluginmirror project, that This is the only real outlier. I might just have to shove it into one of my On Sat, Jan 31, 2015 at 12:15 PM, Mike Hansen [email protected]
|
|
Hello @MikeHansenMe ! Same issue here. It seems repositories are, at some points, automatically emptied. It happened to me lastly with https://github.com/wp-plugins/google-sitemap-generator Though, it wasn't when I added it as a submodule couple days ago. |
|
Hi @xavismeh thanks for letting us know. We just cloned this. It should all be available now. |
|
For future comments, before mentioning your plugin here. Let's check over at pluginmirror.com that it has been cloned. If not click clone, then let us know if it is not cloned after a few hours. |
|
This one is also empty: http://pluginmirror.com/plugins/events-manager https://github.com/wp-plugins/events-manager/ |
|
My confusion is mostly that this was working in the past, at least with the Then it looks like the entire github repo got wiped, and things started But subversion repos with a tagged release but not a trunk release are no On Mon, Feb 9, 2015 at 12:17 AM, Mitar [email protected] wrote:
|
|
@mitar thanks for letting us know. We will get it in the queue today. @jowrjowr You are correct, the whole repo was deleted and we are still working to get everything restored. Currently there are around 500 plugins that the mirror thinks are already mirrored however they are not. The only way for us to get these ones pushed is manually. If you have another plugin that is not showing please leave a link here and we will get it added. |
|
That's the only one that I've personally found to be an issue, however http://i18n.svn.wordpress.org/ Keeping this updated on the client side is a massive nuisance because of On Mon, Feb 9, 2015 at 8:32 AM, Mike Hansen [email protected]
|
|
Hello @MikeHansenMe Thanks |
|
It looks like I was mistaken when I said there weren't any others.
I am an edge case magnet, I swear go god! These appear to all have non-empty subversion trunks, too. These plugins were all properly cloning from the pluginmirror github in the It is, by the way, much appreciated that you are keeping these mirrored Might I suggest a "This is broken, rekick" button on a plugin's page? That On Tue, Feb 10, 2015 at 8:46 AM, Mike Hansen [email protected]
|
|
We added the plugins to the queue to be updated and also added a new button "Update" to force an update. You can now find plugins on plugin mirror and click update. It will still have to go throught the queue but at least users can trigger it. Thanks again for your effort and feedback. |
|
Where is the button? @MikeHansenMe 😃 |
|
Not sure why the button was not showing on that one but it has been pushed out now. |
|
@MikeHansenMe I am seeing a lot of the plugins are empty lately. In order to understand what causing the issue I tried to setup local copy of the project. I have managed to run the app, however I am not sure how the plugins actually get added to the pluginmirror app. So far looking at the code I am did not find much. Am I missing something obvious here? Really appreciate your help. |
|
Hey @azizur the empty plugins is not a result of the mirror. What happened was during the restore process all of the repos were created in a manual way and some of the plugins (~700) think they have been cloned on the mirror and link to github to empty directories. If the link exists on the site but the repo is not on github it is one of the 700 plugins that are in a state of limbo. For each of those we have to push them manually. If you have some specific plugins, please let us know and we will get them updated. |
|
media-category-library is another one: Thanks for all the effort! |
|
I wrote a small python script which automatically checks if a plugin is empty (on the wordpress site), has a github clone that is empty or not import requests
import json
import sys
import time
import calendar
#https://api.github.com/rate_limit
def check_rate( requests, s):
rate_limit_url = "https://api.github.com/rate_limit"
rate_limit = s.get(rate_limit_url).json()
# the "rate" rate limit is the smaller one
if rate_limit["rate"]["limit"] < requests:
print "number requests exceeds limits: " + str(requests) + ">" + str(rate_limit["rate"]["limit"])
exit(1)
# check if the remaining rate limit is sufficient
if rate_limit["rate"]["remaining"] < requests:
wait = rate_limit["rate"]["reset"] - int(calendar.timegm(time.gmtime()))
if wait > 0:
print "wait " + str(wait) + " secs for the rate limit to reset"
time.sleep(wait)
# now check if the resource limit is sufficient
rate_limit = s.get(rate_limit_url).json()
if rate_limit["resources"]["core"]["limit"] < requests:
print "number requests exceeds limits: " + str(requests) + ">" + str(rate_limit["resources"]["core"]["limit"])
exit(1)
# check if the remaining limit is sufficient
if rate_limit["resources"]["core"]["remaining"] < requests:
wait = rate_limit["resources"]["core"]["reset"] - int(calendar.timegm(time.gmtime()))
if wait > 0:
print "wait " + str(wait) + " secs for the resource limit to reset"
time.sleep(wait)
print "rate remaining: " + str(rate_limit["rate"]["remaining"])
print "resource core remaining: " + str(rate_limit["resources"]["core"]["remaining"])
def does_wordpress_plugin_exist(name):
base_url = "https://wordpress.org/plugins/"
plugin_head = requests.head(base_url+name)
return plugin_head.status_code == 302
per_page = 50
all_repos_first_url = "https://api.github.com/users/wp-plugins/repos?per_page=" + str(per_page)
all_repos_url = all_repos_first_url + "&page=1"
# start a session
s = requests.Session()
# authenticate, which increases the rate limit
# s.auth = ('username','password')
while True:
# check if rate allows the number of request (plus a safety margin)
check_rate( per_page + 5, s)
# get the list of repos for the current page
repos = s.get(all_repos_url)
content = repos.json()
# iterate through all current repose
for repo in content:
if not does_wordpress_plugin_exist(repo['name']):
print "empty plugin: " + repo['name']
continue
# get the commits
commits_url = "https://api.github.com/repos/wp-plugins/" + repo['name'] + "/commits"
commit = s.get(commits_url)
commit_content = commit.json()
# if there are no commits
if "message" in commit_content and commit_content["message"] == "Git Repository is empty.":
print "empty repo: " + repo['name']
else:
print "ok repo: " + repo['name']
# get the next page
if "next" in repos.links:
all_repos_url = repos.links["next"]["url"]
else:
break
sys.stdout.flush()there is a rate limit on the github api, so the 's.auth' should be used to authenticate. There is still a lmit of 5000 request (per hour I think), but the script will wait.... The output can be grepped for
|
|
The script finally ran through and unfortunately, it identified 22729 repos that are empty on github... |
|
I'd love to see the public plugins all listed with non-empty repos. In my case, I don't know in advance which plugin I'll have to clone and process, and I want to minimize manual interaction to the bare minimum. |
|
FYI: I got 33162 of 52712 repos reported as empty. Some come with a warning about being unlisted, others not. Cannot attach the list, so here it is (too big for pastebin): http://pasted.co/97d7173d |
|
We're working on the issues currently. There is no need for more examples at this time. We'll have a fix out as soon as possible. |
|
Old version available: https://github.com/wp-plugins/wp-fullcalendar There is already 1.0 out. |
Take this plugin for example: http://www.pluginmirror.com/plugins/wp-post-date-remover
The plugin is still hosted and functional. A github link is even created for it, but it hasn't been cloned.
The text was updated successfully, but these errors were encountered: