As a developer, you will come across several situations when you need to import data from external sources. Sometimes you will need to import data from more than one source into your Drupal 9 website. The Feeds module in Drupal is a simple, no-fuss module that enables even non-technical users to import content from different data sources. What makes it simpler than other Drupal modules like the Migrate module, is that the entire import process is done through Drupal’s UI. It delivers content to you the way you want it as soon as it is available.
Many websites import RSS / XML feeds from third party sites. In this blog, we are going to get a news feed imported through this 3rd party API that provides data in json format with the help of the Feeds module in Drupal 9.
Installing the Feeds Module
The Feeds Drupal 9 module lets you import data from different formats like CSV, XML, JSON, RSS, etc. into nodes, users, and taxonomy terms via the UI.
1.Install the Feeds module (Here we will be using composer)
composer require 'drupal/feeds:^3.0@alpha'
2. Here, we are using JSON feeds for importing data. So, we require a JSON parser which will need us to install one more module which is the Feeds extensible parser. Best way to install this module is by using composer because it downloads all required libraries. (You can also skip the first step if you're downloading using composer as feeds will be a dependency for the feed_ex module).
composer require 'drupal/feeds_ex:^1.0@alpha'
Note : If you're not using composer to install then the libraries required by this module can also be installed with the Ludwig module. Install the Ludwig module and then go to /admin/reports/packages for further instructions.
3. Enable the feeds and feeds_ex (Feeds Extensible Parsers) module either by using drush or through UI.
Configuring and Implementing the Feeds Module
Here, we are going to import the data from https://newsapi.org/ website articles (Topics related to “Top headlines from TechCrunch right now”) which is exposed in JSON format shown in the below screenshot.
To get this news data we need to login to the website and get an API key. Next, select the topics for which you wish to import the data to your website.
Looking at the API, lets create a content type where our data needs to be imported.
Creating a Content Type
Here, I have created a new content type called as “News”. Choose the field type according to your JSON for which data needs to be migrated.
In the above screenshot, we have chosen simple fields for importing the data like Author which is Text(plain) field for the importing data from JSON key author.
As you see in the above screenshot, we have added the Feeds item field of type Feed and chosen the feed type accordingly. We will be learning how to create a Feed type in the next section below.
Creating a Feed Type
Follow the below steps to create a feed type.
- Navigate to Feeds overview page (/admin/structure/feeds). You can see all the feeds listed in this page.
I have already created 2 feeds. Here I am going to be using News feeds. You can create your own feed type by clicking on Add feed type. - When editing feed type or adding new feed type, we will see the below options.
- Basic settings
Let’s configure the Basic settings for the feed type, First, provide the Name, Description and Explanation or submission guidelines for the feed type.
Import period (shown in the below settings tab) helps you choose how often the import should run. This will be done through a cron job. If you select Off then the import will be done manually.
- The Fetcher settings option shows where you’re importing the content or fetching the content from.
It provides 3 options to choose from,
- Directory :- Import the content from a file or a series of files already on your website.
- Download from URL :- Import the content from url (Which we will be choosing in our case).
- Upload file :- Import content from a file.
Based on the Fetcher selected, we will have to configure the Fetcher settings. For the secondary option of fetcher settings, we have the below list of settings.
- Auto detects feeds
- Use PubSubHubbub
- Always download
- Request time out
Configure secondary settings accordingly.
• Parser lets you configure the format of the feed. Here we have many options to choose from like CSV, HTML, OPML, JsonPath (which is our setting), etc. Choose the one from the dropdown menu according to your feed format.
Based on the parser selected, we will have to configure the parser settings. In our case we don’t have any parser settings for parsers of type JsonPath.
• Processor settings show configurations of the type of content the feed will create.
Similarly, we have a secondary settings option for processor as shown in the image below.
Configure your secondary settings according to the requirement.
The below screenshot shows the primary settings of fetcher, parser and processor.
3. Mapping sources to target
- Context
The help text says base query to run, which shows root or base of the data to be targeted to import the feeds as content. In our example, we have configured as “$.articles.*” because our data resides inside the article key. Look at this path documentation to configure yours.
- Field Mappings
Now, we can configure the Feeds to decide which value pulled in from the JSON should map to which field on the Drupal news content type.
These targets are the Drupal data fields and are pulled in based on the processor you select when creating the feed type. You can also configure other settings for the particular mappings of the fields like Language, filter format (applicable for text formatted fields), unique (unique identifier used to track if the content is new or existing one), etc.
For example, if we want to import the JSON feed news article content title as Drupal news content title field then target will be the Drupal Title field and source will be JSON news article title key as shown in below screenshot.
Here we are using the mapping configuration as shown in the below screenshot.
You can also clone the feed type using the Entity clone Drupal module that is also supported in Drupal 9.
Creating a Feed
To create a feed, navigate to content -> Feeds tab -> add feed -> select the feed type
I have already created one for News. To create a new feed, click on the + Add feed button and configure the feed as required. For our blog, we have configured as shown in the below screenshot.
After the configuration, click on Save and Import. This will import the feeds as content. When the data is getting imported, you will see the processing status as shown in the below screenshot.
Once the JSON data gets migrated to content, we can see the content added to the news content type as shown in the screenshot below.
Now, let’s compare the imported content with feeds.
For more detailed information you can refer to this guide.
Additional modules to extend the Feeds module
• Feeds Tamper module helps you modify and update he source data before it gets imported to your website.
• Feeds extensible parsers contains a set of parsers like XPath XML & HTML, JSONPath JSON & JSON lines parser, etc.
• Feeds Import Preview module lets you see the content in your source before importing it.
• Commerce Feeds module helps you in importing products to your Drupal Commerce site.
