There is quite some literature on the topic of Open Source Software quality/maturity/etc. assessment, which could be surveyed. I think the QualiPSo project had dedicated work packages on such topics (EU FP project ;)

I'd also suggest to plan interoperability between ADMS.F/OSS and evaluation methods such as QSOS : http://www.qsos.org/ for instance.

Another interesting resource is http://ohloh.net/ (currently down ?) which provides some (disputable) analysis of the maturity of projects.

The FLOSSmetrics project (flossmetrics.org, 2006), reusing tools such as SLOCcount (Wheeler, 2004), have developed methods to obtain software metrics for GForge-based forges. The former OSOR.EU website had a good implementation of this. Unfortunately, these projects do not suggest a uniform content format to exchange information about software metrics. 

The OMG has produced a UML Metamodel (UMI) and XML Schema to represent software metrics: the Software Metrics Metamodel.

The only issue of course is that such software metrics are not always available on the host forge.

not sure if it fits here: statistics about localization (e.g. Drupal has statistics on how many strings, labels etc are translated into language X, Y, Z...)

I have a few concerns about assessing the size of a software asset based on the CLOC, that is, this is closely related to the issue of nested assets: how do we estimate the size of a large open source asset (ie. JBoss)? Can we assume its size is the sum of the dependencies' sizes plus its own code? It seems a bit confusing as a definition.

I tend to agree with all the rest, ie. # of users/downloads/commits to assess reputation/activity.

I've been looking at the resources people have mentioned here. 

The Qualipso project looks to have ended in 2010 or early 2011 (the 'final newsletter' is dated jan 2011) and I notice that one of the partners is Atos Origin, the company behind QSOS so presumably there's a good bit of re-use there. Looking at the partners in the Qualipso project I know people at several of them (as I dare say do others in the group). Whether they are the right people of course is another matter! However, I will see if I can contact someone who can tell us more about this project as a lot of it looks very relevant to our work.

Ohloh is also interesting. The most revealing page on their site seems to be the API documentation but I can't find anything about their methodology. They do, however, seem to be confident that they can tell you how many lines of code there are in a project - which seems odd to me as Savino's comment on this topic is well made I'd say!

In terms of our vocabulary, my suggestion is that we simply have a class of "Quality Report" which would probably be a literal (or a sub-type thereof such as an XML literal). We can then suggest that projects A, B and C are worth looking for (and Qsos seems to be a good candidate).

In the Proposed conceptual model of February 7, it is suggested to model metrics as a Literal. However, I fear that Literal will not make information about software metrics better exchangeable between software forges. This would require specifying what should be inside such a Literal.

In my view, software metrics are likely to be multidimensional data structures: every measure (e.g. #downloads, #comments) is likely going to be releative to a number of dimensions (e.g. Time, but also other dimensions may apply). I wonder whether we could therefore do the following:

1. Link Software Asset in the conceptual model to a multidimensional data structure, lets call it "Data Set" or "Data Cube".
2. Define in the specification a number of metrics, such as these proposed by Szabolcs.
3. For each metric, define in the specification a number of dimensions according to which data can be made available (it may be allowed to have no dimensions for a metric);
4. When we will produce the RDF/XML Schema, we may consider pointing to existing Data Cube classes such these in the RDF Data Cube Vocabulary. 

I think the probem here is my use of the word 'Literal'.

The text within the Quality Assurance class in says: "A record of a formal assessment of the software according to a recognisable process." What I was trying to convey there is that projects like Qualipso and Qsos produce a report as an XML document that includes the kind of metrics we're talking about. So the class would be manifested as a chunk of XML, not just a string. RDF has a way of including such chunks of XML (XMLLiteral !) and within an XML representation of ADMS.F/OSS this would just be a self contained block - but the current examples are XML either way.

We *could* of course specify individual metrics as suggested - that's not hard and if the WG feels that's what shoud be done, OK, but it seems to me that a better way is to point to the detailed work that others have done and say "use that" ?

Let me start by saying that there are a number of different evaluation methods. In the beginning it was the David Wheeler's GRAM/S (Generally Recognized As Mature/Safe), then the two Open Source Maturity Models (the one by Cap Gemini and the one by Navica), followed by the Business Readiness Rating (designed by the University of Carnegie Mellon, O'Reilly, Intel and SpikeSource) and the above mentioned QSOS.

Some of these methods focus on aspects related to the organisation behind the open source project, other methodologies look into technical and functional aspects. But as a matter of fact none of them has seen an high level of adoption and this probably indicates a flaw in their approaches. Among the possible causes, the most important seems to be the absence of an objective method to measure those metrics.

In fact while few EU-funded projects have created a number of very useful tools to measure some metrics - being FLOSSMETRICS probably the most prolific in this respect - we still miss a methodology that stick to what is measurable in an easy, replicable and objective way. (e.g. 'bug reactivity').

A well defined methodology should measure code, community and processes properties, lowering to the minimum the risk that different people could assign different values to the same metric.

Last but not least, we should make our best to make possible to associate a given report to a specific version of the methodology, since open source evaluation and selection methods keep evolving.
 

Metrics all included in the conceptual model. WG meeting of 14/2/12 seemed happy with the properties recorded.

I believe we should spend more time discussing this, even if we don't want to address how those metrics could be eventually calculated. For example, collecting any useful infomation about the number of installations is hard at best. While some installers do provide automatic mechanisms for that (e.g. bitnami stacks), generally speaking there is no solution to the problem (and automatic surveys can't be used for anonymous users).

So said, I'm more concerned by the lack of some easy to compute and yet useful metrics, e.g. bug responsiveness. If the WG wants to address this issue I'm convinced we should discuss it further.