NeuralDigest: Applying LLMs to News Articles

Introduction

Hi, I’m Liam Pieri, a software developer passionate about applying Natural Language Processing (NLP) to real-world challenges. NeuralDigest, my latest project, is a news summarization platform powered by Large Language Models (LLMs).
I created NeuralDigest to apply NLP against real-world data and refine my development process. The project involved using LLMs like OpenAI’s GPT-3 to create a tool that distills lengthy news articles into concise summaries.

For example with this article about Foxconn losing key iPhone 16 production responsibilities.

Apple is looking to diversify its supply chain away from China by tripling its iPhone production capacity in India within the next two years. Luxshare has been awarded key iPhone 16 production responsibilities, and is expected to ship 45–50 million iPhones in 2023. Foxconn, Pegatron, and Wistron have been instructed to increase their capacity and manpower in India. Luxshare will produce the iPhone 16 Pro Max, Foxconn will produce the iPhone 16 and iPhone 16 Pro, and Pegatron will produce the iPhone 16 Plus.

http://neuraldigest.org.s3-website-us-east-1.amazonaws.com/articles/2023-05-18.html

Working with OpenAI

For NeuralDigest, I opted to use the OpenAI’s GPT-3 due to its impressive ability to understand the context of lengthy texts and generate highly coherent summaries. The primary objective here was to create a pipeline that could take an array of news articles as input and output concise, yet comprehensive summaries.

Here’s the way that I leverage LLMs in my current project.

llm = OpenAI(temperature=0, openai_api_key=config["OPENAI_API_KEY"], model_name="text-curie-001")

chain = load_summarize_chain(llm, chain_type="map_reduce")

return chain.run(texts)

Let me break down what’s happening here:

1. Initial Setup: I start by initializing the LLM with OpenAI. I set the temperature to 0 to ensure that the model generates the most probable output text, which is crucial for creating accurate summaries.
2. Loading the Summarize Chain: Next, I load the summarization chain. This chain prompts the Curie model to generate a summary . The ‘map_reduce’ type chain I used is designed to handle multiple texts that may be longer than the context length, which is ideal for summarizing long news articles.
3. Running the Chain: Finally, I run the summarization chain on the article. This is where the actual summarization happens.

With this setup, NeuralDigest can take any number of news articles and create clear, concise summaries, providing users with the information they need quickly and efficiently.

The Architecture of NeuralDigest

NeuralDigest is a comprehensive project complete from scraping data to creating content. A diagram of the project can be seen below.

• NeuralDigest-Scraper: Scrapes news articles available on Google News RSS Feed, place it in a S3 bucket.
• NeuralDigest-WeaviateETL: Takes the article text and uses Weaviate API to embed with OpenAI and upsert vectors to Weaviate.
• NeuralDigest-LLM: Gather the article text from Weaviate and generates a day summary, uploading the result to a S3 static website neuraldigest.org

Each of these jobs are deployed as Lambda functions, ran once daily.

Results and Feedback

NeuralDigest reduces the size of news articles by about 79.74%. This means that readers are only dealing with approximately one-fifth of the original text, enabling them to save a significant amount of time while still gaining the essential understanding of the news piece.

While this Kolena Article gives strong evidence for why Turbo is the optimal model for summarization. Langchain support for this solution isn’t optimized, resulting in long wait times.

I chose Curie the final model to reduce cost for the project, while maintaining a high BERT_F1 score.

Future Plans for NeuralDigest

The next step for NeuralDigest is to explore more ways that NLP can be applied to this data. The main directions I see myself taking the project are.

1. Create an news informed chat agent capable of answering questions about the days headlines.
2. Topic Modelling over time: Applying UMAP (Uniform Manifold Approximation and Projection) and create a monitoring solution for news topics as they evolve.
3. Extracting structured data using LLMs, for example, generate a knowledge graph.

Conclusion

Developing NeuralDigest has been a rewarding experience, demonstrating the real-world potential of NLP and LLMs. With an average reduction in article size by almost 80%, the results underline the transformative power of AI technologies.

I look forward to enhancing NeuralDigest and exploring more AI-driven solutions. I hope this overview has provided valuable insights into my work. If you have any questions or wish to connect, feel free to reach out.

Thank you for reading, and stay tuned for more updates on NeuralDigest and future projects.