Let's code up the approach we defined previously, as follows (the code file is available as Intent_and_entity_extraction.ipynb in GitHub):
- Import the datasets, as shown in the following code:
!wget https://www.dropbox.com/s/qpw1wnmho8v0gi4/atis.zip!unzip atis.zip
Load the training dataset:
import numpy as np import pandas as pdimport pickleDATA_DIR="/content"def load_ds(fname='atis.train.pkl'): with open(fname, 'rb') as stream: ds,dicts = pickle.load(stream) print('Done loading: ', fname) print(' samples: {:4d}'.format(len(ds['query']))) print(' vocab_size: {:4d}'.format(len(dicts['token_ids']))) print(' slot count: {:4d}'.format(len(dicts['slot_ids']))) print(' intent count: {:4d}'.format(len(dicts['intent_ids']))) return ...