Hugging Face¶

Advanced natural language processing built for Jax, PyTorch, and TensorFlow

🤗 Transformers provides thousands of pre-trained models that support text classification, information extraction, question answering, summarization, translation, and text generation in over 100 languages. Its mission is to make cutting-edge NLP technology accessible to everyone.

🤗 Transformers offers an API for quick downloading and usage, allowing you to apply pre-trained models to given text, fine-tune on your dataset, and then share with the community through the model hub. Additionally, each defined Python module is completely independent, making it easy to modify and quickly research experiments.

🤗 Transformers supports the three most popular deep learning libraries: Jax, PyTorch, and TensorFlow — seamlessly integrating with them. You can train your model using one framework and then load and infer with another.

Online Demos¶

You can directly test most models on the model hub model pages. Hugging Face also offers private model hosting, model version management, and inference APIs.

Here are some examples:

• Fill in the blanks with BERT
• Named entity recognition with Electra
• Text generation with GPT-2
• Natural language inference with RoBERTa
• Text summarization with BART
• Question answering with DistilBERT
• Translation with T5

Write With Transformer, developed by the Hugging Face team, is the official demo for text generation.

Customized Support Services Offered by Hugging Face¶

Quick Start¶

Hugging Face provides a pipeline API for quickly using models. The pipeline aggregates pre-trained models and corresponding text preprocessing. Here’s a quick example of using the pipeline to classify sentiment:

>>> from transformers import pipeline

# Using the sentiment analysis pipeline
>>> classifier = pipeline('sentiment-analysis')
>>> classifier('We are very happy to introduce pipeline to the transformers repository.')
[{'label': 'POSITIVE', 'score': 0.9996980428695679}]

The second line of code downloads and caches the pre-trained model used by the pipeline, while the third line evaluates the given text. The answer "positive" has a confidence of 99%.

Many NLP tasks have out-of-the-box pre-trained pipelines. For example, Hugging Face can easily extract question answers from given text:

>>> from transformers import pipeline

# Using the question answering pipeline
>>> question_answerer = pipeline('question-answering')
>>> question_answerer({
...     'question': 'What is the name of the repository?',
...     'context': 'Pipeline has been included in the huggingface/transformers repository'
... })
{'score': 0.30970096588134766, 'start': 34, 'end': 58, 'answer': 'huggingface/transformers'}

In addition to providing the answer, the pre-trained model also gives the corresponding confidence score, as well as the start and end positions of the answer in the tokenized text. You can learn more about the tasks supported by the pipeline API from this tutorial.

It is also simple to download and use any pre-trained model for your task with just three lines of code. Here’s an example in PyTorch:

>>> from transformers import AutoTokenizer, AutoModel

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
>>> model = AutoModel.from_pretrained("google-bert/bert-base-uncased")

>>> inputs = tokenizer("Hello world!", return_tensors="pt")
>>> outputs = model(**inputs)

Here’s the equivalent TensorFlow code:

>>> from transformers import AutoTokenizer, TFAutoModel

>>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
>>> model = TFAutoModel.from_pretrained("google-bert/bert-base-uncased")

>>> inputs = tokenizer("Hello world!", return_tensors="tf")
>>> outputs = model(**inputs)

The tokenizer provides preprocessing for all pre-trained models and can be called directly on a single string (like in the above example) or on a list. It outputs a dictionary that can be used in downstream code or directly unpacked using the ** expression to pass to the model.

The model itself is a standard Pytorch nn.Module or TensorFlow tf.keras.Model (depending on your backend), and can be used in a conventional manner. This tutorial explains how to integrate such models into classic PyTorch or TensorFlow training loops, or how to use Hugging Face's Trainer API to quickly fine-tune on a new dataset.

Why Use Transformers?¶

1. User-friendly advanced models:

   • Excellent performance in NLU and NLG
   • Educational and practical, with low barriers to entry
   • High-level abstractions, requiring knowledge of only three classes
   • Unified API for all models

2. Lower computational overhead and reduced carbon emissions:

   • Researchers can share trained models instead of retraining from scratch each time
   • Engineers can reduce computation time and production costs
   • Dozens of model architectures, over 2,000 pre-trained models, and support for more than 100 languages

3. Comprehensive support for every part of the model lifecycle:

   • Training advanced models requires just 3 lines of code
   • Models can be easily transferred between different deep learning frameworks
   • Choose the most suitable framework for training, evaluation, and production, with seamless integration

4. Easily customize exclusive models and use cases for your needs:

   • Multiple use cases provided for each model architecture to reproduce original paper results
   • Internal structure of models remains transparent and consistent
   • Model files can be used independently for easy modifications and quick experiments

When Not to Use Transformers?¶

• This library is not a modular neural network toolbox. The code in the model files is intentionally presented in a raw form without additional abstraction, allowing researchers to quickly iterate and modify without getting lost in abstractions and file navigation.
• The Trainer API is not compatible with any model; it is optimized for models in this library. If you are looking for a training loop implementation suitable for general machine learning, please look for another library.
• Although Hugging Face has made significant efforts, the scripts in the examples directory are just examples. They may not be plug-and-play for your specific problem and might require some modifications.

Installation¶

Using pip¶

This repository has been tested with Python 3.8+, Flax 0.4.1+, PyTorch 1.11+, and TensorFlow 2.6+.

You can install 🤗 Transformers in a virtual environment. If you are not familiar with Python's virtual environments, please refer to this user guide.

First, create a virtual environment with the version of Python you plan to use and activate it.

Then, you need to install either Flax, PyTorch, or TensorFlow. For instructions on installing these frameworks on your platform, see the TensorFlow installation page, the PyTorch installation page, or the Flax installation page.

Once one of these backends is successfully installed, you can install 🤗 Transformers as follows:

pip install transformers

If you want to try out examples or use the latest development code before the official release, you need to install from source.

Using conda¶

🤗 Transformers can be installed via conda as follows:

shell script conda install conda-forge::transformers

For instructions on installing either Flax, PyTorch, or TensorFlow via conda, please refer to their respective installation pages.

Model Architectures¶

All model checkpoints supported by 🤗 Transformers are uploaded by users and Hugging Face organizations and are seamlessly integrated with the huggingface.co model hub.

Current number of models: