You can now use the OpenAlex API to filter and group by continents and large geographic regions, such as the Global South. The full documentation is here.

To see a list of institutions in Europe you can do:

https://api.openalex.org/institutions?filter=continent:europe

So simple! You can group by continent as well. This will return a count of works where an author is associated with the institution’s continent:

https://api.openalex.org/works?group-by=institutions.continent

{
  "key": "Q46",
  "key_display_name": "Europe",
  "count": 26968686
},
{
  "key": "Q49",
  "key_display_name": "North America",
  "count": 25175848
},
{
  "key": "Q48",
  "key_display_name": "Asia",
  "count": 24805214
}...

The key field is the wikidata identifier for the continent, such as South America (Q18).

Querying the Global South

The Global South is a term used to identify regions within Latin America, Asia, Africa, and Oceania. We used data from the United Nations to build a list of countries associated with the Global South. It’s available as a boolean filter like:

https://api.openalex.org/institutions?filter=is_global_south:true

This allows for some very cool groupings, such as “show me authors associated with the Global South, grouped by country”:

https://api.openalex.org/authors?filter=last_known_institution.is_global_south:true&group-by=last_known_institution.country_code

New API Filters

We’ve added new filters for works:

• has_pmid – works that have a PubMed identifier
• has_pmcid – works that have a PubMed Central identifier
• repository – works that can be found at the given repository, based on venue ID
• version – works where the given version is available, such as acceptedVersion

Concepts Improvements

As requested by OpenAlex users, we modified the concepts tree so that it is a true hierarchy. This means when you search for works with the concept Computer Science, you’re also getting works tagged with those sub-concepts, such as Artificial Intelligence

We’re excited to announce that we’ve added fulltext search to 57 million articles in OpenAlex, based on data from the General Index. This feature moves OpenAlex’s search function beyond title and abstract, covering the full text of 57 million documents, resulting in ~30 times more search results for many keyword searches!

What is the General Index?

The General Index is a very large database of n-grams that were extracted from 107 million journal articles. It’s openly available without restrictions, and is supported by 100 prominent professors and researchers.

An n-gram is a set of words that occur in a document. For example, in the sentence “the quick brown fox jumped”, a 3-gram would be “quick brown fox” and a bigram would be “brown fox”.

The n-grams from the General Index look like this:

{
    ngram: "sheet of cellulose nitrate",
    ngram_tokens: 4,
    ngram_count: 4
},
{
    ngram: "high than the diameter",
    ngram_tokens: 4,
    ngram_count: 1
}

So we know that the phrase “sheet of cellulose nitrate” occurred in the document four times. The General Index used a tool called spaCy to extract n-grams from articles, capturing from 5-grams down to unigrams from each document.

You cannot recreate a document from these n-grams due to the way that the text was processed (we checked this carefully). However, with the n-grams we know the phrases that exists in each document and how many times it was mentioned… which is great for search!

Enabling fulltext search

We matched the n-grams that had metadata to records in OpenAlex, then loaded the n-grams into Elasticsearch. The result is fine-grained, fulltext search across many articles in OpenAlex. This allows you to find words and phrases deep within a document. This feature is ready to use today!

Fulltext search is integrated into the main search feature, with priority given to title, then abstract, then fulltext: https://api.openalex.org/works?search=dna.

You can filter records to see those that have fulltext available, and you can search fulltext only.

Can I see the n-grams?

Yes you can! Each Work object in OpenAlex now includes an ngrams_url field; the URL you find there that points to a list of that work’s ngrams.

You can also access a work’s ngrams directly via DOI, by using this REST API endpoint:

/works/:doi/ngrams

So for example, to get the ngrams for the work with DOI 10.1016/s0022-2836(75)80083-0, you can call https://api.openalex.org/works/10.1016/s0022-2836(75)80083-0/ngrams.

And the best part is, because these API queries are cached they can be served even more quickly than the rest of our REST API…so you can feel free to scroll through thousands or even millions of DOIs using this endpoint.

Exploring the data

Looking across the OpenAlex data set, about 32% of articles prior to 2000 have fulltext, and about 25% of articles between 2000 and 2020 have fulltext:

The count by year is:

The coverage increases above 50% when a record has a DOI:

Finally, coverage increases above 70% and even up to 80% in years when an article has more than 50 incoming citations:

We hope you enjoy this new feature! We’re thankful to The General Index project for making this incredible data set available, and we’re proud to be one of the first organizations to host it in an easy-to-use manner.

The OpenAlex launch is going well!  Thanks for all of your feedback, comments, questions, and help spreading the word.  A few updates for you below.

Snapshot updates

There is a new native-format snapshot, with the following updates:

• includes “abstract_inverted_index” in works
• includes “raw_affiliation_string” in works.authorships (thanks for requesting this!)
• includes “cited_by_api_url” in works is now a string not a list (sorry! the list was a bug)
• corrected the spelling of institution.associated_institutions
• “ids” dict doesn’t include entries for empty ids anymore (simplifies the data)

This new snapshot doesn’t have additional new works since the previous one, but we expect new works to be added in the next week, and approximately every 2 weeks after that.  A new MAG-format snapshot including new works will also be release at that time.  Each new snapshot will contain articles published up to just a few days before the snapshot release (rather than several weeks old as was the case with MAG). 

API updates

The same changes as described above for the snapshot, importantly including the “abstract_inverted_index” in the list and filter endpoints.

Nature write-up

The OpenAlex launch was covered in Nature this week!  You can read about it here:  https://doi.org/10.1038/d41586-022-00138-y  
We are really happy to hear that people are finding it easy to use!

OpenAlex Tips of the Day

We have been posting tips for using OpenAlex on Twitter every weekday.  

You can see past tips at this search link (whether you have a twitter account or not), and you can follow us on twitter here: @openalex_org

Questions?

We’d love to hear from you: team@ourresearch.org