“Don’t it always seem to go, you don’t know what you’ve got till it’s gone…” Three days ago freecode.com abruptly shut down, claiming “low traffic”, but there has been enough public mourning since to make me pretty sure it fills a need that’s still there. There was nowhere else you could go that was quite so good for getting a cross-sectional view of what the open-source world is doing, independently of any given forge site or distro.
Web frameworks have gotten much more powerful since the original Freshmeat was built 17 years ago; today, I think building a replacement wouldn’t be a huge project. It is not, however, something I am willing to try to do alone. Whether or not this goes forward will depend on how many people are willing to step up and join me. I figure we need a team of about three core co-developers, at least one of whom needs to have some prior expertise at whatever framework we end up using.
In the remainder of this post I’m going to sketch a set of project goals, including both some trimming away of freshmeat/freecode features I thought were unnecessary and some new ideas to address problems in its design.
First, let’s be clear about the problem we’re solving. We want a central place where people can post project release announcements so other people can monitor and search release activity, which also serves as a searchable index of project metadata. The project’s main view is to be a timeline of recent release announcements.
Here are some project goals and constraints I think are important:
1. Avoid moderation overhead.
Freshmeat/freecode required that every project creations and release be pre-moderated by humans. This was a serious bottleneck, and may have been the site’s undoing by imposing staffing costs on the operators. We need to avoid this.
I propose that we can do so simply by having a “report as spam or garbled” link on each displayed announcement. There will still be some human overhead to process these, but experience with social media such as G+ that have this feature suggests that it scales reasonably well.
2. Open-source software only.
Freshmeat/freecode blurred its mission and complicated its job by accepting release announcements for proprietary projects. Let’s not do that.
3. A tool for remote-scripting operations
I hate sites that force me to do clicky-dances on a web interface when the information to be submitted would naturally fit on a job card to be processed by a client. Forge sites that don’t let me remote-script a release action are major culprits – they force me to do irritating hand-work every time I ship.
Freshmeat/freecode had a web-accessible JSON API, and I maintained a freshmeat-submit tool that spoke it. The new site needs to do likewise.
4. Bring back Trove
Free tags are great, and the new site should have them, but I don’t think they’re enough by themselves. I think we lost something very useful when freecode dropped the Trove taxonomy. (Admittedly I may be biased, since I was Trove’s original designer.) The new site should bring back Trove, and have tag folksonomy too, and should use tags as a feeder to the gradual extension of Trove.
5. Simplify, simplify, simplify
In my opinion, Freshmeat/freecode tried to do too much. The “heartbeat” feature, for example, always struck me as pretty useless; nowadays, especially, if I want to view stats on development activity I’ll go to Ohloh. I never saw any good reason to carry links to screenshots; selling the project’s niftiness is what the project’s website is for. There are other features that could have been pared away without loss.
The most important way to hold down complexity is to be careful to specify a clean, simple functional design. Let’s avoid the bells, whistles, and gongs this time.
6. Some thoughts about implementation and other issues.
I’m pretty sure I could get hosting space for the public site at Sunsite.
My #1 candidate for a framework to use is Django. Because Python, and the documentation suggests Django would be well matched to requirements. My mind is open to alternatives.
I’m thinking of this as ‘freshermeat’. We’ll need a better name.
If this sounds like a project you want to sign up for, so indicate in the comments on my blog or G+.
Suggestion: have the projects put their “metadata” into a file in the project repository (something like PROJECT.yaml or whatever). This file should contain all the infos in a somewhat structured format and could be picked up by crawlers / search engines / freshermeat.
Similar to sitemap.xml or robots.txt.
Freshermeat or any other “planet” site could regularly pull this metadata from the projects and update its database / website.
If possible, you want a recent dump of the free code database to seed from, or you may need to scrape the last available copy from the Internet Archive or some other cache … if available. If not, then your going to have a huge hill to climb to establish relevance.
Thumbs-up on Django.
I wish I had time available to commit to working on this project, but I am already dangerously close to overcommitted as it is.
> 1. Avoid moderation overhead.
>
> Freshmeat/freecode required that every project creations and release be pre-moderated by humans.
Project creation might be pre-moderated, release not so much. Another thing: you probably need a way to gather moderators from the user community, a la StackOverflow.
[…]
> 5. Simplify, simplify, simplify
>
> […] I never saw any good reason to carry links to screenshots; selling the project’s niftiness is what the project’s website is for.
*One* screenshot is very useful in selling / describing what project is about. Not every project has a website (be published via e.g. Google Code, or blog, or forum),… though nowadays repository hosting sites make possible to have wiki / project web page / decorated README…
I can’t tell whether the old JSON API was submit only or whether it provided full access to the data aggregated at Freshmeat/freecode. Anyway, I suggest having a web service API for access to the new system’s data so that anyone could slap on a front end, then let the games begin! See this from Steve Yegge: https://plus.google.com/+RipRowan/posts/eVeouesvaVX
Surprised that no one else was amused when you mentioned Django right under a header that basically said KISS. I’m fortunate enough to find jobs that use better languages than PHP to do their stuff but all too often when it’s python they’re using django. It’s made me loath it the more flaws I find in it. Particularly things like their NIH database layer and weird templating syntax. Given your work with docbook Eric I’d think you would prefer chameleon (zope-like valid XML syntax for templates) and django’s model/view setup is just plain irritating. That’s all I’ll say about that.
I don’t want to turn this into a contest for the biggest framework zealot though. So I’ll just say that I recommend something like Pyramid which has a really nice pluggable extension architecture much better than django’s and is the successor of pylons which I’m sure you’ve worked with before. I’m in love with it myself and moving a project over to it that grew out of web.py. However, Django will do the job. It just depends on whether you’re setting out for easiness or robustness.
>I recommend something like Pyramid which has a really nice pluggable extension architecture much better than django’s and is the successor of pylons which I’m sure you’ve worked with before.
Actually, I have no direct experience with any of these, and no conviction about which would be right.
>However, Django will do the job. It just depends on whether you’re setting out for easiness or robustness.
Which do you think is which? That is, which is easy, which is robust?
Talk to Patrick Lenz. He tried to buy Freecode back from Dice. Jeff Covey, who ran the site day-to-day, quit a few days before the closure. I’m sure he’d be willing to help. So would I, and I’m sure Ray Shaw and some of the other Baltimore people would, too.
Perhaps this would be a good project to give newLISP a try?
One question you had that didn’t seem adequately answered before:
newLisp may not have a compiler in the usual sense, but it includes a “link” command that produces a standalone executable file that you can ship.
>Perhaps this would be a good project to give newLISP a try?
Probably not, alas. That would require me to write a web framework, work I’d prefer to avoid.
Rather than a Python framework, I’d suggest something Go related. Speed, ease of maintenance, ease of deployment are all factors Go has that I’ve found too many Python options simply don’t. The style enforcement via gofmt is a real bonus if the code base is going to be appropriately open.
Just a thought. I, too, would enjoy jumping into this, but am probably a bit overextended as is.
This sounds like an interesting project, but I’m not a Python programmer, and my go-to toolset is the Spring Framework, especially for new REST applications. The runtime overhead is larger, but the ease of development is wonderful. How strongly do you feel about Python and/or Django?
>my go-to toolset is the Spring Framework
That would require me to actually learn modern Java, as opposed to merely being able to write small patches in it. While I’m not fanatically opposed to this idea, I’d need to see some serious evidence that Spring is better than any framework in a language I already know before I’d seriously consider it.
>How strongly do you feel about Python and/or Django?
No attachment to Django. Considerable attachment to Python.
I have machines and excess bandwidth which I would be happy to make available, but I’d prefer to work with a PHP base and have an open source framework which is easy to expand and have just moved the front end to Bootstrap3 …
I’d be looking perhaps to include the old commit logging that we used to have ;)
>I’d prefer to work with a PHP base
Sorry, but for a project like this I won’t touch PHP with a 20-foot pole. Too ugly, too hard to maintain.
> Which do you think is which? That is, which is easy, which is robust?
As far as robustness goes I’d say pyramid wins hands down by virtue of not making assumptions about what you’ll be using with it. It can scale from a tiny site that serves a json object (and its builtin way of doing this is just beautiful) all the way up to what you want to do with a complete templating solution and database layer. It also has this /really/ neat way of serving views in which it just returns a dict and no opaque Request type, meaning that you can write unit tests that test the given view like any other python method. I digress though. Point is that pyramid is the one I’d say is robust and I don’t think you’ll have too much trouble with it even if you haven’t touched pylons since these frameworks after a certain point all adhere to certain design philosophies. It basically uses the same model/view pattern that django does. Just better implemented.
On the other hand django is the easiest. It’s pretty much the “do you think there are sane defaults” situation. Django comes bundled up with its own stuff like a DB layer and template interpreter. Pyramid comes as a core for you to build upon that you add extensions to as needed. This can explain it a bit better than I can probably: http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/introduction.html#what-makes-pyramid-unique
Also, look at those dict-returning views and decorator config methods. Is that sexy or what.
But yeah as I said I don’t want to get you hung up on a given framework. It all ends up generating a page anyway. It’s just that from what I’ve seen when it comes to python frameworks, be it ones like django or self-proclaimed microframeworks. This is the one that is truly making an effort and doing a pretty good job of adhering to the KISS principle. So take that as you will.
I’m interested, but I don’t do django. If you were open to doing something in Grails though, I’d *definitely* be down.
If this happens, one thing I’d like to see is good support for DOAP, and maybe FOAF as well. Good search is a must, and I definitely agree about keeping it “open source only”.
>If you were open to doing something in Grails though, I’d *definitely* be down.
Yet another requirement that I learn a new language. Are any two people willing to at least use the same web framework and language going to volunteer?
@Tom – regarding “project metadata” – that is exactly what DOAP is for. A lot of projects do publish DOAP files, but not all of them, unfortunately. And yes, that includes some of my own, which reminds me that I need to correct that ASAP. :-)
http://en.wikipedia.org/wiki/DOAP
Eric, given the gravitas you have in the Open Source world, I’m guessing you are best positioned to try and talk the people who own Freecode/Freshmeat into sharing a dump of the existing data. Any chance you’d be willing/able to reach out to somebody and inquire about that? If not, I’m thinking about starting on a crawler to crawl the site while it’s still up, and try to capture as much data as I can. Unless somebody already has something like that, or we hear (quickly) that they are willing to share data.
>Any chance you’d be willing/able to reach out to somebody and inquire about [a dump of the existing data]?
Sure. I’ve emailed Scoop about this; not sure who else to ask.
Ha, you of all people should know better than to expect that. ;-)
That said, Grails is based on Groovy, which is essentially some syntactic sugar around Java with some of the type safety relaxed. I’ve recently transitioned into writing partially in Groovy, when I prefer the convenience of builder specifications and its handling of some boilerplate, and in Java when I want strict compile-time checking.
Up until recently, Spring was a good framework but required quite a bit of configuration to get bootstrapped. What really tipped the balance for me is the Spring Boot project, which will autoconfigure nearly everything. You can literally write a class that you want stored in a database, drop in a properties file with the database credentials, and have a complete working persistence system set up with nothing else. All of the Web plumbing is also handled transparently, including an embedded server and JSON and XML marshalling.
I particularly like Thymeleaf for doing any HTML work, since its templates are strictly valid, viewable HTML5 with data attributes that indicate where the engine should do replacement. Most of the other template systems I’ve seen (JSP, Mustache) give you gobbledygook if you open them in a browser.
@Christoper Smith – so, would you be open to working on a Grails project for this? If so, that gives Eric two volunteers who are willing to use the same platform. I expect we’d want the code for the site itself to be open source as well? And if that isn’t the plan, I’m out, or will do my own thing.
If anybody is interested, I’m happy to go ahead and spin up a template Grails project, create a Github repo, etc. to give people something to start working with.
Also, I have a couple of domain names, one of which might be useful for this initiative, which I’d be willing to donate to the cause if needed.
> Sure. I’ve emailed Scoop about this; not sure who else to ask.
Rad. I agree with the poster upstream who said it will be hard to establish traction if we are starting from scratch. *Supposedly* somebody crawled Freshmeat a couple of years ago and turned all of the data into DOAP, but I’m not sure if an actual dump is available anywhere or not.
esr>”Probably not, alas. That would require me to write a web framework, work I’d prefer to avoid.”
I know nothing about it, but Google likes Dragonfly
http://dragonfly.neocortex.io/
@Phillip
I’m not nearly familiar enough with the Grails idioms for routing and rendering, and I mostly rely on the Spring MVC magic to handle that for me. With my current load, I don’t think I would be able to spin up learning Grails, especially as MVC since 3.1 has added most of what I understand Grails to do in a manner that makes more intuitive sense to me.
I might be able to work with a service facade and lower layers if Grails were just going to be a wrapper around them. I don’t think Thymeleaf has reliably working Grails support yet, though, and IMO it’s a lot cleaner solution than GSP. Also, I’m not familiar with the current state of Grails Boot or whatever the best Spring Boot integration is, and the autoconfig (especially of the servlet container) knocks out a major disadvantage JVM applications have had against Python and Ruby.
Finally, I generally work with Maven. I don’t have any inherent objections to Gradle, but the Eclipse support even in GGTS is kinda flaky.
@Christopher – understood. Ideally we would compose this in a nice modular fashion with good “separation of concerns” and possibly using a suite of microservices or whatever anyway. I could see a Grails app that is just the “front end” and calling into services written in Java (or Groovy) either as REStful services or in-process.
Just to satisfy my own curiosity, if nothing else, I’m going to spend some time this weekend hacking on a Grails app and try to implement some bits and pieces of this. If it leads somewhere, that’s great, if not – no harm, no foul. The repo is up on Github at https://github.com/mindcrime/freshermeat if anybody else wants to poke at it.
Of course, if the back-end is a set of RESTful services, they could be written in $whatever. Python, Go, Haskell, Erlang, or Brainfuck, it wouldn’t really matter. :-)
I’m wracking my brain trying to remember some sort of open forge hosting software project I had bumped into a while back. No luck so far. I know I’m not thinking of github, which itself is openly released but half-intentionally not made easy to self-host. I do believe there’s another project like it which is made to be intentionally easy to self-host. That’s not what I’m thinking of either.
But if there’s something starting from scratch, I’d love to have my nose to the glass on it. This comment is to subscribe me to this thread.
Software discoverability has already been mentioned, and I like seeing that stressed.
I’d also like to see a strong focus on connecting users to one another and to a project’s developers. Seeing “reviews” on, say, Firefox’s various extensions/addons where some are bug reports, complaints or replies is an example of a problem “solved” just to tick a feature box. A simple issue tracker is probably a good answer.
Someone already mentioned that software projects don’t always have proper hosting (e.g. they use a general-interest personal blog). Such a site could be more than a place to release software, but could also provide some basic hosting. KISS would just be FTP with 10MB of plain HTML/CSS/JS files, and a separate gallery tool for images. Videos can be linked-to and hosted on YouTube, Vimeo, etc.
As for curation.. what if developers were given blogging functionality? Then that same functionality could be given to a select few curators, who would have both overviews and reviews of software. For example, there could be one or more curators of command line utility software. The curators would follow tags, get comments from users and check out software. Then their collection of blogs could be compiled in an RSS feed for interested users.
It’s no longer good enough for me to have a pile of links, to have to check every program out, figure out what it does (hopefully its description is actually descriptive), try it and then throw it away when I don’t like it. Tags are fine if I have lots of time and interest. However, these days I love seeing things like “Top 10 FOSS RSS readers” curated by someone I trust.
I’d love to curate software.
Oh, and it needs to be said that a name should probably be discussed on IRC with the intent to purchase a domain then and there. The notion of throwing around ideas here and then getting around to worrying about a domain is sloppy (and open to sharking).
Conspiracy theory: They were sent an NSL.
The last useful OSS service that went suddenly down with a bogus-sounding explanation was TrueCrypt, and it’s been all but confirmed that the three letter agencies came a-knocking on their doors.
I think there’s enough interest to make this project happen, but getting something resembling a rough spec is going to need to be an initial requirement. Spiral’s mentioned some features that are useful but that I really don’t think should be here; instead of reinventing the wheel and making a big monolithic endpoint, I suggest just having a field for a bug-tracker URL and letting GitHub or JIRA or Bugzilla handle all of that stuff. Same with blogging; that changes this from a lightweight event bus to something resembling more of a big forge site.
> I know nothing about it, but Google likes Dragonfly
> http://dragonfly.neocortex.io/
newLisp has a second web framework as well, called “newlisp on rockets”, by Jeremy Reimer.
http://newlisponrockets.com/rockets-main.lsp
Jeremy used it to implement a blog and forum that are really nicely integrated.
And if the Rockets or Dragonfly frameworks are inadequate, their authors would be very interested and responsive to your feedback.
I’m willing to work in python, it’s my prefered language. I can learn django or pyramid as needed. From here I don’t have an opinion about which. I’ve done some web work in python. Database and backend work is where I live, I doubt that this project will be difficult in that area. Postgresql is my preferred database engine, but most any sql server will do if someone else knows how to set it up and manage it. I could learn, but expect that the web engine will take most of the time and attention I can spare for now.
Jim Hurlburt
Yakima, WA
FWIW, I renamed the repo I started, since it’s probably not likely to become “the” repo, and I don’t want to seem like I’m treading on ESR’s naming idea. Yeah, I know “freshermeat” probably won’t become “the” name either, but still… I’ve changed mine to “sourcehub”
https://github.com/mindcrime/sourcehub
@Christopher Smith
You have a point about requiring a spec.
It feels like a lot of things a software project needs are already found elsewhere, and I hate duplication of effort. However, it also feels like some or all of those “elsewheres” are vulnerable to being suddenly discontinued.
Hell, maybe zero hosting capabilities are actually required and the problem that needs solving may just be the discoverability of software. But then, isn’t that what dmoz is for?
I guess we can go back to esr’s original post on this:
http://esr.ibiblio.org/?p=5936
> With this site gone, where can people go for a cross-sectional view of what open source is doing?
What about simply python without any framework?
Python has enough bells and whistles for something like this tool and you can write clean code too.
If I might dare to suggest, why not Python as a CGI-BIN application? I know CGI is old technology, but is there any real objection to pure python using the simplest of web interfaces?
>If I might dare to suggest, why not Python as a CGI-BIN application?
I considered it, and I’ve done it in the past, but it makes the database stuff kind of painful.
I’d say something like:
Resources: principal (person/organization), project, release. Just a stream of current releases and the ability to look up a specific resource might do most of what’s needed. Perhaps tags or categories that stand alone as resources instead of just being attributes. Longer-term, perhaps some sort of graph queries to identify clustering.
spiralofhope
“I’m wracking my brain trying to remember some sort of open forge hosting software project I had bumped into a while back. No luck so far. I know I’m not thinking of github, which itself is openly released but half-intentionally not made easy to self-host. I do believe there’s another project like it which is made to be intentionally easy to self-host. That’s not what I’m thinking of either.”
I personally like the interface of repo.or.cz. I believe it is even older than github.
http://repo.or.cz/
Why argue about the web interface? The main function would be to collect links to the latest releases in public (git) source code repositories. All the relevant information should be present in the repositories of the participating projects.
Why not divide the work into an online database and a web interface. The DB gets send a commit link by the projects covered from some commit trigger in the individual project’s repositories, e.g, git. Anyone can then build a seperate website for browsing the db by querying the db using a public API. Or make the “Mother” db a git repository and everyone can clone it.
I must admit that my own experiences with such sites are limited to very simple single project sites.
> I’m thinking of this as ‘freshermeat’. We’ll need a better name.
If you don’t use some form of “Bazaar”, either it has to be because you thought of something even cooler, or you should seek immediate medical attention.
@ spiralofhope:
gitorious?
I won’t say that the world couldn’t use another site for Open Source project hosting, but if the goal is just to replace Freshmeat, I don’t think hosting really comes into it – at least not now. Freshmeat was really just a directory and a feed for new releases. I think that’s what we need to create now, modulo the changes esr suggested above, and maybe a few other tweaks. For one, I personally would definitely want to see this site made “semantic web ready” by supporting DOAP and FOAF and what-not. I’d also like to let users edit release notes… it always bugged me that if you made a typo you had to beg a freshmeat admin to fix it for you. :-(
>I won’t say that the world couldn’t use another site for Open Source project hosting, but if the goal is just to replace Freshmeat, I don’t think hosting really comes into it
I agree.
Flask is a pretty good Python web framework if you want the freedom to select your helper packages. I’ve developed web sites with it, and find it useful without getting in the way. Miguel Grinberg’s blog and his O’Reilly book are good starting places, and the Flask documentation is pretty good, too. Django has great documentation and a strong development community, but I believe it suffers from too much mass. Once you select Django, you’re all in. Pyramid seems kind of similar to me in that regard, perhaps because it inherits much from Zope, which also became a massive self contained project.
Depending on what level of querying and mineability is desired, planning for a full-on graph database might not be overengineering at this point. Neo4J is the main efficient graph database that I’m familiar with, and Spring has solid mapping support for it. It’s definitely a niche tool compared to an RDBMS or even MongoDB, but it’s unparalleled for directness of expressing complex relationships and efficiently querying them.
> If I might dare to suggest, why not Python as a CGI-BIN application? I know CGI is old technology, but is there any real objection to pure python using the simplest of web interfaces?
If to go without CMS, or framework, or microframework, then WSGI not CGI.
one thing that made freshmeat inconvenient was the effort it took to announce new releases.
having to remember to do it, find the right place, fill in all the fields, all that always meant that it’s another item on the todo list when i am to busy already.
announcing a new version should be as simple as pasting a link to the download url.
a version can probably be extracted from that url, and if not at least a timestamp can be used (unless the download file is hosted on a braindead server that has no or wrong timestamps)
but even better would be to fully automate it. i think at least debian is making an effort to automatically detect updates and notify packagers.
so that’s something that i’d like to see as a feature.
ideas that come to mind is to watch source code repos and watch for release tags, or at least include the date of the last commit.
even if the last release entry is old, having a recent commit date is telling that the project is still alive and it might be worth checking if there is a newer version available.
and if this is done in some pluggable manner so that the community can contribute detection mechanisms, and then further if an api can be used by distributions to plug into this then this project might attract contributions from distributions as well.
think about it, who is interested in updates? not only end users. actually at least for users of distributions the most interesting updates are in the packages, and less in upstream…
as an end user i am interested in the following:
* find out what applications are available and being able to compare them at least superficially
* see if they are still maintained
* find out if they are available in any distribution
as a linux distributor i am interested in
* the latest version of our packages
* popular packages that we are not including
* alternatives to packages that are no longer under development
as a developer i care about
* easy reporting of releases
* finding updates to tools and libraries i am using
* finding active projects to contribute to
greetings, eMBee.
>one thing that made freshmeat inconvenient was the effort it took to announce new releases.
I solved that problem years ago. Look at freecode-submit and shipper. With these tools, sending a release notification became an easy part of a scripted release process.
The key to this was that freecode had a remote-scriptable API. That’s why it’s on my requirements list.
Python2 or Python3 ;)
The biggest problem these days is that there are perhaps too many languages and variations?
I started with Algol many moons ago and still have some of the punched cards, but switched to C/C++ on leaving the university environment. When I needed to augment the legacy C based client code with a web based system, PHP5 was just nearing release, and was a refreshing change which was a simple learning curve. Current PHP5.x is an utter pain in the arse, but with 90% of my active projects based on it I make it do what *I* want. To support it though I have Eclipse and have to patch Java to keep the add-ons in that working. I’m running Mercurial as the DVCS platform and patches in Python2 to fit that into the framework. One of my data sources is OpenStreetMap which relies on Ruby for its tools, and I have to run Postgres rather than Firebird which I have all of may active data froj the last 20 years.
Add HTML5, CSS3 and javascript and just where do you draw the line :(
I miss a number of tools that have been lost in the last few years, and the replacements provide many more complications which ‘fix’ some contrived problems but at the expense of creating their own idiosyncrasies. Sourceforge, github, bitbucket and the associated project hosting all do things differently enough to make project tracking a minefield, so there IS a place for a central tool that creates a level playing field independent of the hosting platforms and to some extent independent of ‘programming language’
Maintaining a working platform i difficult which ever language you are using, and we all have our own preferences based on past experience rather than perhaps the ‘best’ choice, and it is not possible to simply switch track these days. I probably have a couple of years worth of work simply bringing the current client base up to the current ‘standard’, not because there is anything wrong with the current installations, but simply because others dictate that code used is now ‘end of life’ but the job being done does not change every year. I’m STILL doing the same job on many sites which was done 20 years ago, but I have to pay out on new licenses or re-write core code simply because of someone elses actions.
What will be the fashion next year anybody know yet :)
I do not believe there is a need. Search engines today are extremely good and almost all open-source projects have public version control repositories, complete with web interfaces and descriptions. A lot of them have websites too. Based on these realities, I consider freecode obsolete.
For what it’s worth I got into open source about 4 years ago, so I missed the Freshmeat hype wave (which might just be the reason I am not excited for a replacement).
I placed my own project in FreeCode before it shut down. The driving reasons from my side were the minimal effort to post a page, and the completeness of the database, with hopefully a bit more exposure for my low-popularity code.
I think the “push” submit model is broken. I think that there are really good structured program/version number data out there in the (eg) Fedora and Debian repositories (debian has uscan, fedora now has its own version scraping tools for major websites). Its not 100% complete, but you should certainly be digging into that resource to ensure that FreeCode is up-to-date. Similarly scraping (within legal limits) major repositories, such as SF might be valuable. Talking to perhaps the debian or fedora people to implement common versioning maintenance (each maintainer in debian maintains uscan scripts by hand) might be useful.
The question of how to offer value is a valid one – simple replication of existing pages is not enough. Things like comparison tables are popular, and can be pulled (automatically?) from wikipedia. Having side-by-side feature comparison charts would be nice.
IMHO the traffic was low as simply making a big database of links is not always useful to all. There needs to be something more, I don’t know what.
End users certainly don’t care if you write in python, ruby, or algol. It certainly is not the primary question. As long as it works, and does a job that is appreciated, thats great.
The primary question is “what will be valuable?”
>I think the “push” submit model is broken. I think that there are really good structured program/version number data out there in the (eg) Fedora and Debian repositories (debian has uscan, fedora now has its own version scraping tools for major websites).
I’ve looked at that end of the problem. Scraping most recent commits is easy; scraping releases is hard. The hardest part is how you automatically mine release notes – uscan doesn’t even try to to this, and I’m betting the Fedora tools don’t either. Knowing where to scan is also difficult. For these reasons I think pushes by release managers will remain necessary.
esr:
“Yet another requirement that I learn a new language. Are any two people willing to at least use the same web framework and language going to volunteer?”
I hadn’t realize just how much the Open Source languages and frameworks had fragmented until reading this thread. Do have an issue here that potentially threatens the ability of future OS projects (not just Freshermeat) to attract a reasonable number of interested volunteers and sustain over time?
Say what you will about C, C++, and Java; they all had (and probably continue to have) huge numbers of developers who are completely comfortably with them. I’m not sure that was ever true of any single OS framework.
@Cathy, you are probably not involved in the web development community. C, C++ are almost never used for anything other than the most performance-critical parts of a full-stack web application.
Most webapps — even the largest ones, are built using frameworks almost exclusively. Java is still alive and kicking. In fact, most of the more popular frameworks for it (like Spring) are open source too.
The problem lies in the constantly changing trends. In 2006 it was Ruby on Rails, in 2008 it was Django, in 2011 it was node.js, now it’s Scala + Play2… the skill set is always changing and many frameworks find themselves becoming obsolete for no particular reason.
“The problem lies in the constantly changing trends. In 2006 it was Ruby on Rails, in 2008 it was Django, in 2011 it was node.js, now it’s Scala + Play2… the skill set is always changing and many frameworks find themselves becoming obsolete for no particular reason.”
That probably sums it up nicely. One reason I’m happy with PHP is being able to drop into the C code that it’s written in and tinker which is not so easy with some of the later options :)
> I’ve looked at that end of the problem. Scraping most recent commits is easy; scraping releases is hard. The hardest part is how you automatically mine release notes – uscan doesn’t even try to to this, and I’m betting the Fedora tools don’t either. Knowing where to scan is also difficult. For these reasons I think pushes by release managers will remain necessary.
The best way to solve that end is to devise a standard format for publishing that information (I’m thinking in an XML document with a defined schema) and a URI that can be pushed once to the Repobase (which is itself simply a database of these URIs). Then each distro can subscribe to the RSS feed for that application, and/or “poll” for new documents periodically.
For example, https://www.catb.org/~esr/shipper/bazaar/current.xml would have the information about the current release of shipper, and would be a link to https://www.catb.org/~esr/shipper/bazaar/1.3.xml. The overall history of the project could be imputed from the simple directory listing https://www.catb.org/~esr/shipper/bazaar/
>The best way to solve that end is to devise a standard format for publishing that information (I’m thinking in an XML document with a defined schema)
Hah. Good luck (a) getting enough projects interested to make the effort, (b) persuading them to use a markup as heavyweight as XML, and (c) relying on them to get the markup right. By the time you’ve passed these three filters you’re in the land of statistical noise. I speak from grim experience here, having spent 13 years attacking very similar problems in documentation formats.
>>I think the “push” submit model is broken. I think that there are really good structured program/version number data out there in the (eg) Fedora and Debian repositories (debian has uscan, fedora now has its own version scraping tools for major websites).
>
>I’ve looked at that end of the problem. Scraping most recent commits is easy; scraping releases is hard. The hardest part is how you automatically mine release notes – uscan doesn’t even try to to this, and I’m betting the Fedora tools don’t either. Knowing where to scan is also difficult. For these reasons I think pushes by release managers will remain necessary.
This would mean that *only* release managers would have reasonable chance to keep information in Freshermeat up to date.
We have now various visual scrapper helpers, for example Kimonify (proprietary service) or Portia (Python), visual scraping for Scrapy (Python). One solution would be configuring scraper when creating repository (for web page, or mailing list, or IRC channel).
Another solution would be providing webhooks for repository hosting sites, like GitHub, Gitorious, BitBucket or Google Code (I don’t know if all of them support web hooks).
BTW. I wonder if it wouldn’t be a good idea to add some of functionality of now defunct CIA.vc…
I really don’t think that the tool in question should directly include any sort of scraper or pull functionality; it simply isn’t needed. GitHub and other repository services have Web hooks and triggers, and services such as Travis already manage a subset of this issue entirely based on those postbacks. If scraping really turns out to be required, make it a standalone program that runs a scraping job and posts the results to the new service.
>If scraping really turns out to be required, make it a standalone program that runs a scraping job and posts the results to the new service.
Agreed. This is proper separation of function.
> Hah. Good luck (a) getting enough projects interested to make the effort, (b) persuading them to use a markup as heavyweight as XML, and (c) relying on them to get the markup right. By the time you’ve passed these three filters you’re in the land of statistical noise. I speak from grim experience here, having spent 13 years attacking very similar problems in documentation formats
a) Fine. If they don’t want their project to have any visibility, I can’t make anyone make the effort, but it seems like no more work than filling out some form on a web page at Freshermeat/Bazaar.
b) I’m not married to XML. You can use JSON or whatever if you think it works better. I mention XML because the toolkit to use it is ubiquitous, and the schema mechanism provides help with validation.
c) Fine. If they can’t be bothered to write up the metadata for their project correctly, what confidence should I have in their code?
“@Cathy, you are probably not involved in the web development community. C, C++ are almost never used for anything other than the most performance-critical parts of a full-stack web application.”
I was actively involved in the web development community (not Open Source) 1999 – 2000. Things have changed a bit since then. :-)
My point was meant to be broader than that. If you wanted to write applications, in, say, 1993 – 95, it was pretty clear what languages you needed to learn. But this thread is telling me that if you want to write web apps today, it’s not at all clear what you should invest your time in learning.
“Most webapps — even the largest ones, are built using frameworks almost exclusively.”
Yes, I’d expect that given the complexity of modern web apps. Why reinvent the wheel?
“The problem lies in the constantly changing trends. In 2006 it was Ruby on Rails, in 2008 it was Django, in 2011 it was node.js, now it’s Scala + Play2… the skill set is always changing and many frameworks find themselves becoming obsolete for no particular reason.”
And this was the real point I was trying to make. Pre-web, developer skills needed didn’t change all that quickly. But reading this thread has made me realize that if I wanted to get back into web development, it would not be at all clear what it’s worth my time to study, and what not. And it looks to be very difficult to get a project started when practically every person who expresses has a [different] framework, and sometimes even language, to suggest.
If you wanted to get into free software/open source, in the 90’s it was clear you needed to learn C. This mean there was a common ground for open source projects. What happened to that common ground?
> The best way to solve that end is to devise a standard format for publishing that information
> (I’m thinking in an XML document with a defined schema)
That’s pretty much what DOAP is. Of course, that’s not to say DOAP is either perfect or complete, but it’s probably at least a good start.
> GitHub and other repository services have Web hooks and triggers,
Yeah, this is what I was thinking as well. As long as their is a remote API for publishing releases and we can integrate with Web Hooks or whatever, then that reduces (at least some) of the need for any notion of “scraping” for project data.
That said, as someone who is admittedly a proponent of the Semantic Web ideal, I wouldn’t be opposed to something that can crawl “DOAP Space” and pull in information. But that does open up more than one can of worms, and could probably be punted until “version 2” or something. :-)
>>If I might dare to suggest, why not Python as a CGI-BIN application? I know CGI is old technology, but is there any real objection to pure python using the simplest of web interfaces?
If going that route, I strongly suggest mod_python. It hooks into apache pretty well and is low overhead. From there db work is pretty easy.
Jim
> That said, as someone who is admittedly a proponent of the Semantic Web ideal, I wouldn’t be opposed to something that can crawl “DOAP Space” and pull in information.
If only there were something crawling the Web to make it easy to search for files named
doap*.rdf. Someone really ought to work on that. A service like that could make managing a site like Freshermeat (as described) trivial, because it could have pre-composed queries designed to find such files, rather than having to host them all. Maybe it could help people create properly formatted DOAP files via some sort of web form.>If only there were something crawling the Web to make it easy to search for files named doap*.rdf
Separation of function, dude. You build the crawler, someone else will build the suite with an API the crawler can use to ship what it finds.
> The hardest part is how you automatically mine release notes
do you mean to mine the actual content of the release notes, or to find the link for a release announcement?
i think the most important is to find out about the existence of a new version. if that can be automated then a few things are possible:
an email could be sent to project maintainers to remind them to share the relevant details.
(a reply could be processed automatically)
notifications could also be sent to volunteers,
and anyone coming to the site can be asked to help fill in missing details too.
greetings, eMBee.
>do you mean to mine the actual content of the release notes, or to find the link for a release announcement?
I mean the actual content of the release notes. But just mining for a version is pretty nasty, too.
Let me tell you how I know this. Eleven years ago I started working on shipper, which has to have a version and release notes to fully do its job. (The release notes used to be needed for the freshmeat/freecode announcement; they’re still needed for automatic generation of email notifications.)
Early versions of shipper used a bunch of different methods to try to automatically mine the version number from the released sources. They were complicated, ugly, and failure-prone, so much so that I eventually gave up on the whole idea; now, it’s the responsibility of the user to pass the version number to shipper. Shipper still mines for current release notes; its ability to do so depends on the project having a NEWS file that follows some unsurprising conventions.
>an email could be sent to project maintainers to remind them to share the relevant details.
This, and all following suggestions, are exactly diametrically the wrong direction to go. Automatic data mining, good when it can be made reliable; adding hand-work, baaad. Very bad. Produces errors, missed steps, unreliable data.
In the interests of KISS, this project should stick to announcing version update, along with the changelog/release notes and a link to the project site in the format of a blog, with automated submission through API calls. Further classification of update can be either MAJOR release, MINOR release or BUG FIXES.
At the project database level, the following details may be provided on creation of project;
1. project name + description
2. project author email (hidden or obfuscated from public view of course)
3, download link (repository)
4. project icon / logo – optional
5. project website – optional
After that, the ability to view updates in either chronological order or sorted projectwise.
>In the interests of KISS
This is almost exactly what’s in my (slightly incomplete) design sketch.
I forgot to add “project category’ to the above to allow easy searching of project based on what kind of application the user is looking for.
> I never saw any good reason to carry links to screenshots; selling the project’s niftiness is what
> the project’s website is for.
Perhaps I’m an unusually visual person, but in the case of an app with a GUI, I positvely *love* seeing a screenshot up-front. You can tell a lot about how much care and attention is put into a GUI at a glance.
> >an email could be sent to project maintainers to remind them to share the relevant details.
> This, and all following suggestions, are exactly diametrically the wrong direction to go. Automatic data mining, good when it can be made reliable; adding hand-work, baaad. Very bad. Produces errors, missed steps, unreliable data.
i am not quite following your argument here. not that i have any better answers, i am just trying to understand what you mean.
obviously we agree that automation is good when it can be made reliable. but how does adding hand-work make it bad? sure, human error can always break things, but the only way out of that is either moderation (which is expensive) or better automation (which is difficult)
are you suggesting that no automation is better than bad automation?
that may be true, so the next question i’d like to ask is what can be automated in a way that won’t produce bad data?
my idea of mining for new versions is to simply try and find out if there is a new version at all.
there are 3 possible outcomes:
nothing is found – which is no worse than not mining at all.
a new release is found – at which point an email to the developers and volunteers (people that previously agreed to help, and who can be moderated until proven trustworthy if needed) should result in correct data, or no answer. if there is no answer the release is ignored as it could be a false-positive
a false-positive is found – again an email to developers or volunteers should produce a correction.
if alerting those that have an interest in good data does not work, then how do you expect updates to work at all?
one could argue that notifications don’t improve things and are therefore not worth implementing. i don’t have the experience to answer that, but you seem to suggest that they’d make things worse. so i am wondering what you mean here.
note that i am not talking about notifications to casual visitors who just want to know if there is an update. those should only be sent reliable data, of course.
greetings, eMBee.
>are you suggesting that no automation is better than bad automation?
I am suggesting that any metadata-update process that launches on a heuristic and then requires multiple stages of human attention is doomed. People just won’t do it, and when they do they’ll make mistakes. Better to have the whole release metadata update be either automatically generated or (where that can’t be relied in) the responsibility of one release manager.
This is how shipper is designed, In the normal case, it gets all its per-project data by either (a) parsing a control file that doesn’t change across releases, (b) parsing the NEWS file for release notes, or (c) the release manager supplying a version when it’s called.
> Early versions of shipper used a bunch of different methods to try to automatically mine the version number from the released sources. They were complicated, ugly, and failure-prone, so much so that I eventually gave up on the whole idea; now, it’s the responsibility of the user to pass the version number to shipper. Shipper still mines for current release notes; its ability to do so depends on the project having a NEWS file that follows some unsurprising conventions.
What about convention that Git uses, namely RelNotes symlink to e.g. Documentation/RelNotes/2.1.0.txt (in AsciiDoc-like format)?
>What about convention that Git uses, namely RelNotes symlink to e.g. Documentation/RelNotes/2.1.0.txt (in AsciiDoc-like format)?
It’s a reasonable convention, but only Git uses it. You’re attacking the problem from what I think is the wrong end, trying to effectively mandate a very specific metadata format that thousands of projects would have to do fiddly work to switch to in order to be visible to the new site. This is not an approach that has been successful in the past.
In designing shipper’s metadata-mining logic, I took the opposite approach. I looked at existing, well-established conventions and figured out how to parse those. Take a look, for example, at the NEWS file of galaxis. Thousands of projects already have NEWS files that look almost exactly like this. So I wrote the following rules:
1. Get the contents of NEWS or HISTORY.
2. Skip to the first blank line, then skip blank lines. (This discards a title header)
3. Skip the first nonblank line. (This discards a per-release header line)
4. Interpret all lines up to the next blank line (but not including it) as the current release notes.
In general, the less you prod people to alter their existing metadata practices – and the better case you can make that you are merely asking them to conform to what a majority or at least plurality of projects already do – the better your chances of getting them to do what you want are.
Also, I think it is usually ineffective to try to prescribe strictly structured metadata formats; many people consider these an imposition. It’s better to do permissive parsing (cycles are cheap) and not expect people to make more than small changes to conform to it.
Why hasn’t anyone suggested using the old tried-and-true LSM format or an extension thereof?
>Why hasn’t anyone suggested using the old tried-and-true LSM format or an extension thereof?
In fact, the job cards freshmeat-submit processed resembled extended LSMs.
Given Python, I reckon
Flask
http://flask.pocoo.org/
or Web2Py
http://www.web2py.com/
are both excellent low overhead/low boilerplate frameworks.
I have used web2py for this mini-project:
http://acosta.pythonanywhere.com/
and am impressed with its flexibility and ease of use.
Thanks for spearheading this effort. Freshmeat/Freecode, as well as the even better AppWatch site that used to exist years ago, have always been very valuable resources to me as a Linux user, sysadmin, and distro developer.
I do really hope you won’t give up on the idea of automating finding new releases. Depending on projects to submit something is going to miss a lot of projects. This was what made AppWatch so great back in the day, so much better than Freshmeat, even as great as it was. It successfully found new versions and release note blurbs and did it for years until I think ZDNet bought it and eventually shut it down.
The big thing I would say — freecode got a lot less useful when they went from fielded data and structured and hierarchical data to tags. Talk to any library and information sciences person — structured and validated indexing is way better than a pile of tags.
Also check the competitions. “alternativeto.net” is one with a little different approach.
Hey gang, I’ve knocked out a modest amount of functionality around a Freshmeat like site already, in my Grails based app. I see a lot of people talking about Python, but so far all I see is talk, and no code. :-)
If a group coalesces and moves forward with a Python based version, that’ll be great. In the meantime, if anybody knows Groovy – or is a good Web Designer – and would like to pitch in and work on this Grails version, the code is at: https://github.com/mindcrime/sourcehub
I’ve put everything I’ve done so far under the Apache License v2.
>I see a lot of people talking about Python, but so far all I see is talk, and no code. :-)
Shipping code is a nearly unanswerable argument. :-) I’ll take a look.
Is shipper “pluggable” e.g. wrt. how release notes are written (c.f. Dist::Zilla in Perl/CPAN)?
>Is shipper “pluggable” e.g. wrt. how release notes are written (c.f. Dist::Zilla in Perl/CPAN)?
Not sure what this means. Explain?
Regarding the work I have done: Note that I am *not* a designer, so there’s nothing visually appealing about the pages, and yes, it is using Bootstrap. :-)
My goal is to get basic functionality in place (create project, list projects, create release, and the front-page “release stream”) and then come back and flesh in more functionality, as well as getting a real designer to “make it pretty”.
So far that codebase has “create project” (where a “project” is just name and description at the moment), user login, ability to list all projects, ability to list projects you own (eg, that you created) and not much more. My plan is to implement “releases” and the front-page stream tonight or tomorrow, and then start fleshing in more details for a project: links, tags, trove categories, etc., etc.
>> Is shipper “pluggable” e.g. wrt. how release notes are written (c.f. Dist::Zilla in Perl/CPAN)?
>
> Not sure what this means. Explain?
Is shipper easily extensible with plugins? Dist::Zilla is a build / authoring tool, extensible and configurable with plugins, with a rich set of plugins. For example shipping to CPAN or NPM, or creating a DOAP file, or building an RPM, or Twitter-ing a release, or getting version from script, is a plugin.
>Is shipper easily extensible with plugins?
Not yet. That would fit the architecture well, though.
I’ve been doing web development with Django for the last five years or so off and on (much more so “on” in the last nine months), and I’m interested in volunteering to work on this project.
I’ve recently worked on creating an JSON API with the tastypie Django app, which can be pretty drop-dead simple depending on what exactly you need it to do and how closely it’s tied to your models. There are plenty of other possibilities for REST APIs with Django, as well, of course.
I’m not a hacker my any means (and I’ve never worked on an open source project before), but I would love to help with this.
What do I need to do to get involved?
@Philip Rhodes – Just a minor concern. Although you’ve changed the name of your project from “freshermeat” to “sourcehub”, the README still says: “Create a database named ‘freshermeat‘ in Postgres”. Is that correct?
> You’re attacking the problem from what I think is the wrong end, trying to effectively mandate a very specific metadata format that thousands of projects would have to do fiddly work to switch to in order to be visible to the new site. This is not an approach that has been successful in the past.
If you can get people to submit project metadata to Freshermeat in whatever format you choose, perhaps via a web interface, then you ought to be able to get that interface to generate a DOAP/LSM/whatever file.
The thing you’re completely ignoring about trying to replace Freshmeat is that whatever reason it isn’t there anymore might apply to Freshermeat too (National Security Letter demanding you provide information you won’t be willing to give up, so your only alternative is to shut the site down?). If, however, you build a site that helps people get their metadata into the right format, then Google, Bing, Yahoo, etc. will provide competing indices, such competition acting as a control should anyone become a bad actor and try to censor certain project info. There’s a fair chance that if you get a good robust metadata format, someone at Google will use some of his mandatory play time to work up tools for it to make searching for project metadata even easier.
If you stop thinking about “building a better Freshmeat”, you can instead build a site designed not to be a single point of failure, distributing the metadata on each project’s own web site, but indexed by search engines that are already doing that for you anyway. So when you get that NSL demanding information you never collect (because your metadata generator is written in Javascript and downloaded to the client, so that you never hold the data yourself) you can politely tell the government that information never reaches your servers.
Will some developers refuse to fill out your metadata? Sure. Invoke the Billingsley Doctrine (from the scene in Airplane where Barbara Billingsley translates Jive to English and back): “Chump don’ want no help? Chump don’ get no help!”
>The thing you’re completely ignoring about trying to replace Freshmeat is that whatever reason it isn’t there anymore might apply to Freshermeat too (National Security Letter demanding you provide information you won’t be willing to give up, so your only alternative is to shut the site down?). I
Um, how could this happen? By design, every bit of information coming in is things release managers want to advertise.
>There’s a fair chance that if you get a good robust metadata format, someone at Google will use some of his mandatory play time to work up tools for it to make searching for project metadata even easier.
That’s putting the cart before the horse, though. The main purpose of Freshmeat was its timeline view.
>If you stop thinking about “building a better Freshmeat”, you can instead build a site designed not to be a single point of failure
Alas, in order for it to fulfil its design purpose (the timeline view) there needs to be a single point of aggregation.
@TM
Thanks for introducing me to the Billingsley Doctrine! :-)
@Jorge – Yeah, I didn’t get around to changing the default database name yet. For right now, if you fire up the app, it will look for a “freshermeat” database.
esr> Alas, in order for it to fulfil its design purpose (the timeline view) there needs to be a single point of aggregation.
Might I suggest that, if Monster’s suggestion has merit here, then it might be possible to perform BOTH of these functions. Perhaps phase one would be coming up with a solid metadata file that could be accessible via HTTP or FTP from the repo that houses the project, then phase two would be building “freshermeat” to perform aggregation by finding these metadata files and display that data in interesting ways.
The “find” function could have two methods, one would be for the maintainer to submit the URL where the metadata file is maintained, the other could be by using Google, et al. to perform discovery.
And of course, a web-page could be created to assist the maintainer in creating the initial metadata file (perhaps even allowing scanning of the various well-known files to create a starting point). I can see this functionality becoming an integral part of github, etc. in the future.
This would do a couple of things:
1. “freshermeat” would be able to tell when a project vanishes and could send notification to the maintainer that there’s a problem (allowing the maintainer to gracefully remove the project, cutting down on cruft in the DB, or make adjustments when he/she changes code repositories).
2. Reduce the overhead somewhat hosting provider (in tern reducing hosting costs – TANSTAAFL :^).
3. Allow other projects to emulate “freshermeat”‘s seek function via Google (ensuring that the efforts of maintainers are not lost if “freshermeat” bites the dust in a couple of years). Also would allow people to come up with some potentially cool new applications for that data that compete or supplement “freshermeat”.
4. Puts a bright-line separation between the data provided by the maintainer and “freshermeat” (addressing Monster’s NSL scenario as well as any issues with software patents, etc.).
Either way, you’re going to have to come up with some schema by which maintainers will have to submit their metadata. Why not do that as a stand-alone, parsable file.
My thought was that, if it was designed correctly, this could be the basis for a tool to automate the creation of packaging system packages. If sufficient information was in the metadata file, then it might be possible to automatically generate .deb or .rpm, or whatever files from the information and the images. Alternately, something like “yum” could be built that would maintain these applications in their own directory tree, providing the glue to tie them into the system (updating paths, adding config changes to /etc, adding icons to desktops, links to files, etc.) on a distro-specific basis. It probably wouldn’t work out-of-the-box for large, complex systems (e.g. large web apps, database engines, window managers, etc.) but for the tons of simple little web apps, desktop apps, and CLI apps, it could make those tools a lot more accessible to the noobs.
That would go a long way to helping the user community AND the dev community and save a LOT of time all around (I hope :^).
Just a thought.
>Perhaps phase one would be coming up with a solid metadata file that could be accessible via HTTP or FTP from the repo that houses the project, then phase two would be building “freshermeat” to perform aggregation by finding these metadata files and display that data in interesting ways.
I think you and the Monster are making the same fundamental mistake here – confusing two different projects.
One project, the one I’m thinking about, is an aggregation point for release announcements – in effect, a feed anyone can monitor (and yes, it should have RSS capability). That should be fed through a remote-scripting API and web-form submission. This is ‘freshermeat’. It’s the direct replacement for the capability we just lost.
It would be nice to have a crawler that generates updates via the API from watching a collection of websites, and a metadata format designed to make life easier for the crawler; this is where DOAP enters.
But these are different projects. By lumbering freshermeat with the requirements for the crawler, and bogging down in what would undoubtedly become a metadata design war (because I’ve looked at DOAP and it’s just not good enough yet) all you’d be assuring is that neither part got done. Please stop that right now.
>Either way, you’re going to have to come up with some schema by which maintainers will have to submit their metadata. Why not do that as a stand-alone, parsable file.
Yes. I’ve already solved this problem. Look at freshmeat-submit and shipper; look at the formats I designed for their metadata files. There is no reason not to reuse much of these designs, they are proven by a decade of field experience.
>My thought was that, if it was designed correctly, this could be the basis for a tool to automate the creation of packaging system packages.
Good insight, but not quite on the mark. Shipper’s per-project metadata reuses (and compatibly extends) the designs of Debian control files and RPM specfiles. The goal is that no piece of metadata that naturally lives in these files should have to be duplicated anywhere else. In fact, in the normal case one of these files is the *only* per-project data shipper needs – there is no project-specific file that is for shipper’s use only.
Why not just use the FSF software directory:
https://directory.fsf.org/
I’m probably guilty of causing some of this by even mentioning DOAP, but I agree with esr, at least in the short-term: I think we should focus on a straight-up replacement for Freshmeat that does no more (or little more) than Freshmeat did, and perhaps less in some areas.
Now, if we get that going, and the site gains traction, and people are motivated to build this other functionality then, hey, I’m all for it. I’ve already said I’m a pro-Semantic Web vision guy (I also vote Libertarian, so you know I’m an idealist), but I’m also pragmatic at times. Personally, I’d like to just get the basics in place first, see if people even still really want a Freshmeat-like, and then let the rest come as it may.
Eric:
In case I wasn’t clear enough in the two posts I made, if you do this in python, I’m willing to spend some serious time in development.
I’ve been on the road on a motorcycle trip this last week, so while I’ve read most of the posts I haven’t attempted to give design of the project any serious thought. I do agree with the goal to keep it focused and simple at the start.
I do mostly backend work, with a quite a bit of experience with glue layers for different systems. I do user interfaces, but cannot call myself any sort of expert on implementing a web based interface. The part to accept a scripted submission, once I’ve figured out how to capture the data from the scripted session should be a piece of cake.
I can probably set up searches by mapping them to a parameterized sql statement(s) although I’m sure that somebody has figured out the good method of handling this problem.
Feeding the data set back to the user should be an easy task.
I don’t mind the notion of django or pyramid or …
I’ve used mod python, and given a set of html templates it wouldn’t be any real trick to create the interface between the user and the data store.
I don’t require much of the above except using python X as the main language, I’m willing to consider learning most any reasonable web framework.
Jim Hurlburt
Yakima, WA
>In case I wasn’t clear enough in the two posts I made, if you do this in python, I’m willing to spend some serious time in development.
I didn’t miss that; thank you, Jim.
Here is the state of play:
1. I am working on a design document for the new project based on my extensive prior experience with the Freshmeat/Freecode API. While I’m not certain I wrote the only client for it ever, I never heard even rumor of another one.
2. Phillip Rhodes has begun an implementation in Grails. I’ve looked at the code but haven’t run it yet; I will, later this week. Shipping code is a powerful argument, but I have some doubts that the maturity and feature set of Grails is sufficient. In particular, if we’d have to code the handler for the remote-scripting API by hand (as opposed to having it be generated in a provably correct way by introspection on the data models) that’s a crash landing and Grails is out.
3. I have been looking af Django, which will generate the API by introspection (there are a couple of extension packages for this, I’ve been looking at tastypie) and seems adequately powerful in other ways. It is a plus that Python and Django are very well documented.
4. I have yet to evaluate Pyramid/Pylons.
So, no Python project has yet been started. It might not happen if Grails is good enough – which at the moment I give about 30-40% odds (they will go up sharply if I learn that there is a Grails extension analogous to tastypie).
If a Python-based project does happen, I’ll bring you in.
If Grails runs into trouble, the Spring 4 handling will be more than sufficient for the described use case, and it won’t be especially difficult to translate; Spring MVC is perfectly fine with controllers in pure or mixed Groovy. What specifically are you concerned about when you say “code the handler”, for both terms?
>If Grails runs into trouble, the Spring 4 handling will be more than sufficient for the described use case, and it won’t be especially difficult to translate; Spring MVC is perfectly fine with controllers in pure or mixed Groovy.
You make me nervous when you say things like that. I’m very sensitive about long-term maintenance costs, and a mixed-language project tends to send them way up. You probably know (and if you don’t, you should) that bugs tend to nucleate around that kind of fracture line in the architecture.
>What specifically are you concerned about when you say “code the handler”, for both terms?
We’re going to need code that can accept requests in JSON or XML (I prefer JSON) to do remote-scripted operations on project and release objects – query, create, modify, delete. For this project, this capability is essential for two reasons (1) I must be able to write a functional replacement for freshmeat-submit, and (2) we want to allow Monster and others who feel passionately about the crawler-centric option to do their thing. They write the DOAP-aware crawler, it exercises the API, all is happiness.
What I want is for the web framework to have a code generator that looks at the structures in the model and generates our request handler. Django plus tastypie does that – push a button, get a REST API. If there’s no equivalent facility in Grails, that implies a lot of hand-hacked handler code that’s going to be a huge time suck and defect attractor. Not good.
Hi,
my last post did not appear and I don’t know if it get verified. So I have put it at my discussion site here http://www.opensourcestore.org/viewtopic.php?f=7&t=6
Lothar
>Why not just use the FSF software directory
I’m uncertain whether you’re being serious or not. I’m going to assume that you are. The answer manyfold:
1) because it’s not representative enough.
Quoting https://directory.fsf.org/wiki/Main_Page
“The Free Software Directory is a catalog of useful free software that runs under free operating systems. FSF staff and volunteers work together to collect detailed information about free software programs and organize it in a clear and accessible format”
Free software is a subset of open source software. The successor to Freemeat/Freecode should have representation from more than the slice of open source software which fits the ‘free software’ label.
2) Because it is curated by hand
ESR argues (unless I’m sorely mistaken) that the need for human intervention may have contributed to Freecode’s death. Even if not, it would kill this initiative. The Directory depends on human intervention. Quoting from https://directory.fsf.org/wiki/Free_Software_Directory:Participate
“Anybody can create an account and update or submit new entries to the Directory. However, updates and new submissions need approval by an administrator before they will appear publicly. Logged-in users can tell if they are seeing the latest version of the page, or if there are newer versions of the page waiting for approval by a message at the top of the page.”
3) No submission API
Although you might be able to POST the data into the submission form, it’s not supported as a submission method. A web API is a must-have for this project — because it helps keep everything automated.
In this case, I don’t think more so than you’d get with mixed C and C++.
Spring MVC does this all implicitly; you specify what class you’re expecting in the request body as a method parameter, and it will get mapped in (and it can actually content-negotiate multiple incoming formats if you have some reason to need both JSON and XML). There’s literally zero configuration required to make that happen with Spring Boot. I’m not as familiar with Grails, but it’s a Web presentation layer that’s written over the rest of the usual Spring infrastructure.
I wonder how people used Freshmeat / Freecode.
I have used it in two ways. First, to search for a project to fill a given need; here Trove based filters were very helpful (e.g. free license + linux + GUI). Second, to view fresh updates – perhaps I would notice some interesting project.
There were also editorials / articles like the one about make replacements.
>Spring MVC does this all implicitly; you specify what class you’re expecting in the request body as a method parameter, and it will get mapped in (and it can actually content-negotiate multiple incoming formats if you have some reason to need both JSON and XML).
This is actually a feature of JAX-WS and JAX-RS (Java API for XML WebServices|RESTful webservices), and JAXB (Java Architecture for XML Binding, which actually helps support any kind of markup through extension) not specifically Spring. If this gets done in java (with or without Spring), I’d love in.
In any case, I could help implementing SOLR for searches, when the time is ripe.
@The Monster
>If you don’t use some form of “Bazaar”, either it has to be because you thought of something even cooler, or you should seek immediate medical attention.
“SourceSouq” ?
“BitSouq” ?
zeke
KISS, but also KILL (Keep It Low Level).
Make sure it’s possible to write a bash script to do the whole job
of using a link from your web pages all the way to a runnable app you
can trust, installed on your own system. Of course you can’t control the packagers, but good methodology should attract adherents.
For trust, I’d like links derived from hashes, e.g.,
web-visible descriptive name here
It should be easy to automate the creation of this kind of alternate
links to package files. This would also support automatic checking
against a blacklist, and a warnlist to warn of gotchas in the
associated package. These lists could be discussed on forums, and
be signed by responsible person. They should be in simple text
format as produced by sha1sum or md5sum, to support the automated
checking/warning process. Voted-useful commentary could be collected
in html pages in a server directory with hash-containing automatically derived names like http://some.domain.org/by-hash/sha256-the_hash_hex.txz-black.html. Or ..tgz-warn.html.
Mentioning the hash part in any blog or wiki should make discussion
easily searchable, though I could see trolls and fudsters lowering
the general signal/noise ratio of search results not constrained
to your site.
The idea is that right-click and copy-link will put the url with
hash kind and value in the clipboard where it’s available to a
script one can invoke with a single click. The script can use wget
and automatically check the hash, create a build directory etc, and
ask if you want to run the package’s own script(s). By that point
you should have seen black or warning list comments and be forewarned.
Also, IMO no one should be asked just to “sudo …” without even
knowing if they’ve actually got the identical package version whose
install directions they are following. And you shouldn’t have to
decode make files and script options to know what persistent changes
will be made where when you hit Enter (or click go).
Those things can usefully be exposed in wikis or blogs, and if
references to packages are …/by-hash/… then it’s unambiguous
what’s being referred to. Anyone can reliably find a copy and unpack
it with a click or two. And it doesn’t matter if the file is
mirrored in a trash bin. It’s verifiable, if you believe the hash.
The /by-hash/ is inspired by linux /dev/disk/by-uuid/… but
let’s not use uuid, rather the forms from sha1sum and md5sum etc.
with sha1- or md5- etc prefixed.
Identifying content by hash is not a new idea either, obviously.
This is all rnr-OTTOMH-stuff, but I hope there might be something useful in this.
If I may add some humor to this thread (with apologies to Deep Purple):
They shut down Freecode one day;
it died without any sound.
ESR is looking for some aid
from the hackerly crowd.
And it won’t be over
’til Linux folks build a new place
to get a cross-sectional
view of the open-source maze.
Smoke from the barrel
of Eric Raymond’s gun!
Smoke from the barrel.
>If I may add some humor to this thread (with apologies to Deep Purple):
Not bad. But the rhyme works better if you replace “maze” with “space”.
For the record, I have never been to Montreux. Nor have I ever burnt anything down with a flare gun. :-)
>But the rhyme works better if you replace “maze” with “space”.
Haha, yeah. But characterizing the open-source space as disorderly was part of my impish intent, as was my poking fun at your reputation as a “gun nut” (a bit of self-mockery there, for I use Linux and support the RKBA).
My threadjacking officially ends here.
I would be interested, and I have bandwidth and rackspace available, as well as a number of spare servers. I don’t use G+.
>I would be interested, and I have bandwidth and rackspace available, as well as a number of spare servers
That’s good to know, thanks.
A few hours ago I had almost no time to explain: I think that the search for the site should be provided by a pluggable service. Hence my proposal to use SOLR; it’s possible that you’ve not heard of it. It’s a highly configurable search engine written in Java and handles requests and responses using open standards. Requests can be channeled with JSON and XML; responses similarly can be given in JSON, XML or in a number of formats.
I’ve implemented SOLR before for a beastly load of input.
In any case, for this application it’d be easy to keep a reasonably up-to-date (within ten minutes of real time) search index with not too much delay. Dislodging the search service from the main application has many benefits: the site provides a beautiful submission API and browsing interface, while the search engine provides potent discoverability features (auto-correction, suggestions, facets [which are filters simmilar to those found on the left-hand panel in amazon product searches]).
It would be good to start work on this aspect as soon as the database model is even moderately settled, and some searchable fields are defined; on the freshermeat app, three features would be neccessary to use SOLR:
1) the search UI.
2) a communication module to send XML/JSON queries and receive XML/JSON responses. This could be as simple as sending GET requests.
3) a hook to send the newly approved/newly sent data to SOLR. This could be as simple as sending a POST request.
I’ve already been working on a new FM substitute over the weekend. I just set up a prototype now:
http://freshcode.club/
While of course rewriting it in any framework would allow more features, it’s not overly much code to get the basics workings. I think the already discussed points about licensing and inversion of the submission process are more crucial. Most importantly I’d demand all entries to be CC-BA-SA licensed. (While the FM database is still available, Dice probably won’t allow any rescue operation; even though the originative copyright holders are the project authors).
Instead of DOAP or any other XML scheme, I’d envision more a “releases.json” to eschew the effortful manual release process. See also: http://fossil.include-once.org/freshcode/wiki/Autoupdate
Though I’d also like to hear if the various Freshmeat API submission tools were in widespread use.
>http://freshcode.club/
That is impressive! What’s it written in?
>Though I’d also like to hear if the various Freshmeat API submission tools were in widespread use.
I don’t know of any other than my freshmeat-submit, and I have no idea what its use volume was. Nonzero – I did get a few bug reports.
@foo
Why not just use the FSF software directory:
Because the FSF is full of moralistic jackasses who don’t want to be contaminated by the foul and impure mojo of open source.
Spring MVC is an alternative to Jersey; not sure about the relative popularity, but they’re both pretty easy to work with. My preference is for Spring because Spring Boot makes deployment so trivial, but they’re both well-supported.
I’m aware of Solr and have used it indirectly but have not worked with it myself. I’m actually involved on a project where it might be very useful and would not be averse to learning it.
Eric and I had an email exchange about this, but just to share knowledge with everybody: Grails does have support for JAX-RS and it’s pretty easy to implement remote APIs over HTTP using JSON or XML in Grails. The ability to create an API won’t be a limiting factor in the Grails version.
Outside of that, Grails is actually pretty mature. The project has been around for several years now, and is ultimately built on top of proven components like Spring and Hibernate. But you get the advantage of the nicer Groovy syntax, which eliminates a lot of Java boilerplate shit you have to deal with when using Java directly.
Was the freecode/freshmeat platform itself not FLOSS? If not, why?
(for my part, any web stuff I do is AGPL licensed).
hi! Just commenting to be suscribed. I used to read freshmeat / freecode, and I miss it
A quick-loading, easy-read, bloat-free freshmeat thing would be to my taste. Low-income and thus frugal, this old man still uses dial-up. Freshmeat/freecode came up reasonably fast with the ‘dillo’ browser. Less pleasant when using ‘firefox’, say. Quickness matters. Slashdot may be headed toward kinda-heavy, lots of gratuitous external links. Guess that I’m both old and old-school. Keep it simple and all that. (Simple and attractive often go hand in hand.)
> What’s it written in?
@esr Run away screaming, but http://freshmeat.club/ (fixed the mod_security issues btw) is enitrely in PHP. While I also prefer the ease of Python nowadays, for simple string crunching and web apps, PHP of late is quite tolerable.
The automated project updates are also almost done – for instance the kernel 3.15.2 entry came straight from regexing and their git changelog. But for broader reusabilty http://fossil.include-once.org/freshcode/wiki/releases.json specifies an ad-hoc format, with which project maintainers or packaging tools can automate release announcements.
FM/FCs data scheme was quite similar, so I can emulate the submit API certainly; http://web.archive.org/web/20130412231142/http://help.freecode.com/kb/api-7/data-api-releases
Except for the withdraw_release() probably..
>Run away screaming, but http://freshmeat.club/ (fixed the mod_security issues btw) is enitrely in PHP.
Bletch. I don’t like PHP at all. On the other hand, you are way ahead of the Grails prototype on features and documentation. And – I admit it – I was independently thinking about “freshcode” as a name. I hadn’t copped to this in public because I didn’t want someone else scarfing up all the available freshcode.* domain names. Your having grabbed one is a significant plus.
>FM/FCs data scheme was quite similar, so I can emulate the submit API certainly
The closer you come to mocking the existing API, the easier it will be to adapt the existing, well-tested freedode-submit tool to work with the new site. An easy adaptation of that would be a good thing.
>Except for the withdraw_release() probably..
Is there some intrinsic problem with that?
@esr: So where’s the git repo for this project? Where do I sign up? I have considerable Django and jQuery expertise to lend, and I’ve even written a JSON REST API.
>@esr: So where’s the git repo for this project? Where do I sign up? I have considerable Django and jQuery expertise to lend, and I’ve even written a JSON REST API.
There isn’t any Django-based project yet. There are two partial implementations, one in PHP and one in Grails. The PHP one has a demonstration site at http://freshcode.club/; the Grails one has a repo at https://github.com/mindcrime/sourcehub.git
Hi, as a regular user I’d like to see a freshmeat replacement as soon as possible. I was shocked with the news of freecode’s demise. I used it in two ways: one, to get informed about new releases of software I cared about; and the other, to read announcements and get informed about new and interesting software projects. Just this very month of June, few days before freecode’s closure, I discovered a *great* product that helped me a lot in my productivity (http://kanboard.net). Without freecode, I feel I cannot “discover” new and great software products. Ohloh is fine, but I can’t see how this functionality is there. Thanks!
General +1 here, I’m not a web dev but willing to donate to help a project get off the ground.
I’m looking at django myself. I am doing some python consulting work besides my legal work and my client’s new project involves django. It’s definitely not easy to pick up for somebody who’s used to thinking low level, but you do get used to it by and by, and it offers some amazing features and power with much less coding.
I think it’s biggest inflexibility is that it doesn’t apparently offer a built-in way to edit data models once you’ve created the database. I must investigate it further.
hari: There are solutions to that. The third-part app south is the de facto standard. If I’m remembering right, Django 1.7 is going to have the functionality built in.
Thanks. It’s good to know that Django 1.7 will have that feature. I am tempted to move my website to django except that my current host has no django support and I am unwilling to move my hosting from my present one.
I’ll miss Freshmeat / Freecode. I only had one small project on the site (BEWorld) but it was one of the primary sources for my Linux distro (LACLIN).
My Front End is limited to Bootstrap 3, some jQuery, and Rails. However, maybe I can help with related CLI tools. Possibly Perl tools; I’ve worked in the language for about 23 years.
My resume is on my site at http://oldcoder.org/ If the project moves forward, feel free to write.
A potential replacement for Freecode.com (freshmeat.net) is going on-line at App-Trove.com
It is in beta testing mode for now, but you can do some things with it already. I particularly like the browsing feature better than the prior site.
Check it out.
There’s now preliminary Trove map support on http://freshcode.club/tags
And we had significantly elevated new submissions due to a bit of open source press coverage this week.
More interestingly, I was able to replicate the http://fossil.include-once.org/freshcode/wiki/Freecode+JSON+API
It’s mostly compatible, but I significantly simplified the URLs updating code, which also reduced some overhead in freecode-submit
http://fossil.include-once.org/freshcode/ci/aa105661b897e7edc01ef418cda27319ed668cae?sbs=1
Albeit I had to later patch it for Python3 and requests.request, as urllib2 on Debian systems still does not provide for TLS with SNI. (Main site uses CloudFlare, so there’s an api. dubdomain now for HTTPs requests).
http://fossil.include-once.org/freshcode/ci/3ffaf89706cdbb20c9b1a9b7817ae840aad04943?sbs=1
It’s pretty much working already.
I don’t know if it’s just me or if everyone else encountering issues with
your site. It appears as though some of the text on your posts are running off the screen. Can someone else please provide feedback and let me know if this is happening to them as well?
This might be a problem with my browser because I’ve had this happen previously.
Kudos