speech to text using cnn

The Captable

Social Story

Enterprise Story

The Decrypting Story

Daily Newsletter

By providing your information, you agree to our Terms of Use and our Privacy Policy. We use vendors that may also process your information to help provide our services. This site is protected by reCAPTCHA Enterprise and the Google Privacy Policy and Terms of Service apply.

Founder first

Announcement

Startup Sectors

Women in tech

Entertainment

Art & Culture

Travel & Leisure

Curtain Raiser

Wine and Food

Speech-to-Text using Convolutional Neural Networks

Thursday April 25, 2019 , 5 min Read

Deep Learning beginners quickly learn that Recurrent Neural Network (RNNs) are for building models for sequential data tasks (such as language translation) whereas Convolutional Neural Networks (CNNs) are for image and video related tasks. This is a pretty good thumb rule - but recent work at Facebook has shown some great results for sequential data just by using CNNs.

In this article I describe my work for using CNNs for Speech-to-Text based on this paper here . I have also open-sourced my PyTorch implementation of the same paper.

RNNs and their limitations

Quick recap - RNNs process information sequentially, i.e., they make use of the sequential information present in the data where one piece of information is dependent on another, and they perform the same operation on each and every element of the sequence. This property of RNNs enables a network to represent complex dependencies between elements in a sequence which is pretty useful for tasks such as speech recognition.

Fig 1: How RNN unrolls when the sequence is passed to it. Here, ‘ x ’ denotes input and ‘ o ’, the output.

However, this advantage comes with two limitations. These dependencies make the model unwieldy as every step is dependent upon the previous operation - so calculation of nodes cannot be divided across processors and have to be done in a sequential manner. This also makes training and inference of the RNN based models a little slow. RNNs also restrict the maximum length of the sequences as all representations then have to be equal to the largest input sequence.

Using CNNs for sequences

CNNs are routinely used for image-related tasks but two papers, which can be found here & here , extended their utility- as encoders for sequential data.

In the first paper the CNN encodes the information from the voice-features. This data is sent to a convolution layer with kernel-size of 1 which also acts as a fully connected layer. This finally gives softmax probabilities of each character that can be placed in the transcription

In the second paper the CNN encodes the information from the image and sends that data to the RNN based decoder which decodes the data and outputs the corresponding text from the image.

This suggests that Convolutional Network can perhaps replace Recurrent for speech-to-text tasks as well.

CNN alleviates both the limitations of using RNNs. Since the input to CNN is not dependent on previous time step calculations can be broken down and done in parallel. This makes training as well as inference much faster compared to the RNN based models. As a CNN learns fixed length context representations and stacking multiple layers it can create larger context representation. This gives the CNN control maximum length of dependencies to be modeled.

A final advantage of using CNNs is that we are much more familiar with them owing to our large body of work in image and video analytics.

Using CNNs for Speech to Text

A speech-to-text pipeline consists of a front-end that processes the raw speech signal, extracts feature from processed data, and then sends features to a deep learning network.

The most widely used frontend is the so-called log-mel frontend also called MFCC, consisting of mel-filterbank energy extraction followed by log compression, where the log compression is used to reduce the dynamic range of filterbank energy. But we used PCEN for our implementation as it led to better accuracy for us. This is explained in detail here .

Output of this frontend usually feeds into some RNN based encoder-decoder architecture which converts these features into corresponding transcription. We used Wav2Letter implementation in NVIDIA OpenSeq2Seq to replace RNN with a CNN architecture.

Fig 2: A fully convolutional network for speech to text.

Nvidia’s implementation was in TensorFlow, which is a great framework, but, bracing the wrath of TF lovers, I dare say I prefer PyTorch - primarily because I find the PyTorch framework more intuitive and easy to run experiments in.

Digging through the internet we found no similar implementation in PyTorch. So we decided to implement the Wav2Letter in the framework ourselves.

We have also open-sourced the entire code developed by us. You can find it here - https://github.com/silversparro/wav2letter.pytorch . As mentioned above, we have replaced MFCC-based front-end with PCEN. Please do check out the open-source implementation and feel free to contribute further.

Accuracy results

This model currently gives the same accuracy as a RNN-based model but with 10-fold decrease in training time and a similar reduction in inference time. This is a work in progress and we shall keep updating the blog with the results of our experiment.

References : Wav2Letter , CRNN , DeepSpeech2, PCEN, AGC , Nvidia wave2Letter , Wav2Letter-lua

Silversparro Technologies aims to help large enterprises solve their key business problems using expertise in Machine Learning and Deep Learning. Silversparro is working with clients across the world for Video Analytics, Computer Vision, Voice automation use cases working for manufacturing, BFSI, healthcare verticals etc. Silversparro is backed by NVIDIA and marquee investors such as Anand Chandrashekaran (Facebook) , Dinesh Agarwal (Indiamart) , Rajesh Sawheny (Innerchef) etc.

Silversparro is founded by IIT Delhi Alumni – Abhinav Kumar Gupta , Ankit Agarwal and Ravikant Bhargava and is working for clients such as Viacom18 , Policybazaar , Aditya Birla Finance Limited , UHV Technologies etc.

• artificialintelligence
• MachineLearning
• Artificial-intelligence-research
• speech-recognition
• neural-networks

MOST VIEWED STORIES

What causes people to live double lives

Stanza Living acquires student housing startup YourShell

With zero cash burn and profit from day one, The Souled Store writes a new e-commerce story

Newly rebranded Ola Consumer resumes cab share rides

A Deep Convolutional Neural Network-Based Speech-to-Text Conversion for Multilingual Languages

• Conference paper
• First Online: 31 March 2022
• Cite this conference paper

• S. Venkatasubramanian 17 &
• R. Mohankumar 17  

Part of the book series: Advances in Intelligent Systems and Computing ((AISC,volume 1420))

744 Accesses

1 Citations

Designers have been processing speech for decades for a wide variety of applications, from mobile communications to automatic reading machines, among others. By eliminating the need for alternative communication methods, speech recognition saves time and money. In the world of electronics and computers, speech is rarely employed because of the complexity and variety of voice signals and noises. Today's technologies allow us to process speech signals quickly and accurately and recognize the text. A real-time translation of speech into written language requires specific techniques, as it must be extremely rapid and nearly error-free to make sense. A person's speech is the most natural and important form of communication. This system converts human speech into a string of words using the speech-to-text (STT) technology. This system's goal is to extract, classify, and recognize information about speech in a variety of ways. Using convolutional neural networks (CNNs) for voice classification, the proposed system is developed. The input signals are classified by CNN on its own since it is a self-optimizing neural network. In addition, high-level features are extracted by convolutional and pooling layer, where the data is classified using fully connected (FC) layer. A database contains pre-recorded speech. Testing and training are the two key aspects of the database. In the training phase, samples from the training database are run through a series of tests to determine their characteristics. Each sample's features are combined to create a feature vector that is stored for future reference. When a sample is supplied to the system for analysis, its features are extracted. There is a comparison between these features and the reference feature vector, and the words with highest similarity are output. MATLAB (V2018a) environment is used to design the system.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or eBook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime
• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Convolutional Neural Network-Enabling Speech Command Recognition

KALAKA-3 Database Language Classifier Through Convolutional Recurrent Neural Network

Real-Time Speech Recognition Using Convolutional Neural Network

Reddy, B.R., Mahender, E.: Speech to text conversion using android platform. Int. J. Eng. Res. Appl. (IJERA) 3 (1), 253–258 (2013)

Google Scholar  

Sultana, S., Akhand, M.A.H., Das, P.K., Rahman, M.H.: Bangla speech-to-text conversion using SAPI. In: 2012 International Conference on Computer and Communication Engineering (ICCCE), pp. 385–390 (2012)

Trivedi, A., Pant, N., Shah, P., Sonik, S., Agrawal, S.: Speech to text and text to speech recognition systems—a review. IOSR J. Comput. Eng. 20 (2), 36–43 (2018)

Ajami, S.: Use of speech-to-text technology for documentation by healthcare providers. Nat. Med. J. India 29 (3), 148 (2016)

Manikandan, K., Patidar, A., Walia, P., Roy, A.B.: Hand Gesture Detection and Conversion to Speech and Text. arXiv:1811.11997 (2018)

Dutta, K., Sarma, K.K.: Multiple feature extraction for RNN-based assamese speech recognition for speech to text conversion application. In: 2012International Conference on Communications, Devices and Intelligent Systems (CODIS), pp. 600–603 (2012)

Sharma, N., Sardana, S.: A real time speech to text conversion system using bidirectional Kalmanfilter in Matlab. In: 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI), pp. 2353–2357 (2016)

Shivakumar, K.M., Aravind, K.G., Anoop, T.V., Gupta, D.: Kannada speech to text conversion using CMU sphinx. 2016 Int. Conf. Inventive Comput. Technol. (ICICT) 3 , 1–6 (2016)

Lakra, S., Prasad, T.V., Sharma, D.K., Atrey, S.H., Sharma, A.K.: Application of Fuzzy Mathematics to Speech-to-Text Conversion by Elimination of Paralinguistic Content. arXiv:1209.4535 (2012)

Kaur, E.J., Nidhi, E., Kaur, M.R.: Issues involved in speech to text conversion. Int. J. Comput. Eng. 512–515 (2012)

Chauhan, V., Dwivedi, S., Karale, P., Potdar, S.M.: Speech to text converter using Gaussian mixture model (GMM). Int. Res. J. Eng. Technol. (IRJET) 3 (5), 160–164 (2016)

Manoharan, J.S.: Capsule network algorithm for performance optimization of text classification. J. Soft Comput. Paradigm (JSCP) 3 (01), 1–9 (2021)

Article   Google Scholar  

Smys, S., Haoxiang, W.: Naïve Bayes and entropy based analysis and classification of humans and chat bots. J. ISMAC 3 (01), 40–49 (2021)

Bapat, A.V., Nagalkar, L.K.: Phonetic speech analysis for speech to text conversion. In: 2008 IEEE Region 10 and the Third international Conference on Industrial and Information Systems, pp. 1–4. IEEE (2008)

Heracleous, P., Ishiguro, H., Hagita, N.: Visual-speech to text conversion applicable to telephone communication for deaf individuals. In: 2011 18th International Conference on Telecommunications, pp. 130–133. IEEE (2011)

Radha, N.: Video retrieval using speech and text in video. Int. Conf. Inventive Comput. Technol. (ICICT) 2 , 1–6 (2016)

Dutta, K.K., GS, A.K.: Double handed Indian sign language to speech and text. In: 2015 Third International Conference on Image Information Processing (ICIIP), 374–377. IEEE (2015)

Dissen, Y., Goldberger, J., Keshet, J.: Formant estimation and tracking: a deep learning approach. J. Acoust. Soc. Am. 145 (2), 642–653 (2019)

Nasib, A.U., Kabir, H., Ahmed, R., Uddin, J.: A real time speech to text conversion technique for Bengali language. In: 2018 International Conference on Computer, Communication, Chemical, Material and Electronic Engineering (IC4ME2), pp. 1–4. IEEE (2018)

Vinnarasu, A., Jose, D.V.: Speech to text conversion and summarization for effective understanding and documentation. Int. J. Electr. Comput. Eng. 9 (5), 3642 (2019)

Nugroho, K., Muljono, M., Marutho, D., Murdowo, S.: Mobile app for word recognition and visualization of objects using Indonesian language google speech to text for deaf students. In: 2020 International Seminar on Application for Technology of Information and Communication (iSemantic), pp. 137–141. IEEE (2020)

Yang, L., Li, Y., Wang, J., Tang, Z.: Post text processing of Chinese speech recognition based on bidirectional LSTM networks and CRF. Electronics 8 (11), 1248 (2019)

Hasan, H.M., Islam, M.A., Hasan, M.T., Hasan, M.A., Rumman, S.I., Shakib, M.N.: A spell-checker integrated machine learning based solution for speech to text conversion. In: 2020 Third International Conference on Smart Systems and Inventive Technology (ICSSIT), pp. 1124–1130. IEEE (2020)

Bano, S., Jithendra, P., Niharika, G.L., Sikhi, Y.: Speech to text translation enabling multilingualism. In: 2020 IEEE International Conference for Innovation in Technology (INOCON), pp. 1–4. IEEE (2020)

Tang, Y., Pino, J., Wang, C., Ma, X., Genzel, D.: A general multi-task learning framework to leverage text data for speech to text tasks. In: ICASSP 2021–2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6209–6213. IEEE (2021)

Passricha, V., Aggarwal, R.K.: Convolutional Neural Networks for Raw Speech Recognition. IntechOpen (2018)

Download references

Author information

Authors and affiliations.

Department of Computer Science, Saranathan College of Engineering, Trichy, Tamil Nadu, 620012, India

S. Venkatasubramanian & R. Mohankumar

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to S. Venkatasubramanian .

Editor information

Editors and affiliations.

Department of ECE, RVS Technical Campus, Coimbatore, Tamil Nadu, India

Departamento de Engenharia Mecanica, Faculdade de Engenharia, Universidade do Porto, Porto, Portugal

João Manuel R. S. Tavares

Faculty of Engineering, Aurel Vlaicu University of Arad, Arad, Romania

Valentina Emilia Balas

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Cite this paper.

Venkatasubramanian, S., Mohankumar, R. (2022). A Deep Convolutional Neural Network-Based Speech-to-Text Conversion for Multilingual Languages. In: Smys, S., Tavares, J.M.R.S., Balas, V.E. (eds) Computational Vision and Bio-Inspired Computing. Advances in Intelligent Systems and Computing, vol 1420. Springer, Singapore. https://doi.org/10.1007/978-981-16-9573-5_44

Download citation

DOI : https://doi.org/10.1007/978-981-16-9573-5_44

Published : 31 March 2022

Publisher Name : Springer, Singapore

Print ISBN : 978-981-16-9572-8

Online ISBN : 978-981-16-9573-5

eBook Packages : Intelligent Technologies and Robotics Intelligent Technologies and Robotics (R0)

Share this paper

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

We apologize for the inconvenience...

To ensure we keep this website safe, please can you confirm you are a human by ticking the box below.

If you are unable to complete the above request please contact us using the below link, providing a screenshot of your experience.

https://ioppublishing.org/contacts/

• Español – América Latina
• Português – Brasil
• Tiếng Việt
• TensorFlow Core

Simple audio recognition: Recognizing keywords

This tutorial demonstrates how to preprocess audio files in the WAV format and build and train a basic automatic speech recognition (ASR) model for recognizing ten different words. You will use a portion of the Speech Commands dataset ( Warden, 2018 ), which contains short (one-second or less) audio clips of commands, such as "down", "go", "left", "no", "right", "stop", "up" and "yes".

Real-world speech and audio recognition systems are complex. But, like image classification with the MNIST dataset , this tutorial should give you a basic understanding of the techniques involved.

Import necessary modules and dependencies. You'll be using tf.keras.utils.audio_dataset_from_directory (introduced in TensorFlow 2.10), which helps generate audio classification datasets from directories of .wav files. You'll also need seaborn for visualization in this tutorial.

Import the mini Speech Commands dataset

To save time with data loading, you will be working with a smaller version of the Speech Commands dataset. The original dataset consists of over 105,000 audio files in the WAV (Waveform) audio file format of people saying 35 different words. This data was collected by Google and released under a CC BY license.

Download and extract the mini_speech_commands.zip file containing the smaller Speech Commands datasets with tf.keras.utils.get_file :

The dataset's audio clips are stored in eight folders corresponding to each speech command: no , yes , down , go , left , up , right , and stop :

Divided into directories this way, you can easily load the data using keras.utils.audio_dataset_from_directory .

The audio clips are 1 second or less at 16kHz. The output_sequence_length=16000 pads the short ones to exactly 1 second (and would trim longer ones) so that they can be easily batched.

The dataset now contains batches of audio clips and integer labels. The audio clips have a shape of (batch, samples, channels) .

This dataset only contains single channel audio, so use the tf.squeeze function to drop the extra axis:

The utils.audio_dataset_from_directory function only returns up to two splits. It's a good idea to keep a test set separate from your validation set. Ideally you'd keep it in a separate directory, but in this case you can use Dataset.shard to split the validation set into two halves. Note that iterating over any shard will load all the data, and only keep its fraction.

Let's plot a few audio waveforms:

Convert waveforms to spectrograms

The waveforms in the dataset are represented in the time domain. Next, you'll transform the waveforms from the time-domain signals into the time-frequency-domain signals by computing the short-time Fourier transform (STFT) to convert the waveforms to as spectrograms , which show frequency changes over time and can be represented as 2D images. You will feed the spectrogram images into your neural network to train the model.

A Fourier transform ( tf.signal.fft ) converts a signal to its component frequencies, but loses all time information. In comparison, STFT ( tf.signal.stft ) splits the signal into windows of time and runs a Fourier transform on each window, preserving some time information, and returning a 2D tensor that you can run standard convolutions on.

Create a utility function for converting waveforms to spectrograms:

• The waveforms need to be of the same length, so that when you convert them to spectrograms, the results have similar dimensions. This can be done by simply zero-padding the audio clips that are shorter than one second (using tf.zeros ).
• When calling tf.signal.stft , choose the frame_length and frame_step parameters such that the generated spectrogram "image" is almost square. For more information on the STFT parameters choice, refer to this Coursera video on audio signal processing and STFT.
• The STFT produces an array of complex numbers representing magnitude and phase. However, in this tutorial you'll only use the magnitude, which you can derive by applying tf.abs on the output of tf.signal.stft .

Next, start exploring the data. Print the shapes of one example's tensorized waveform and the corresponding spectrogram, and play the original audio:

Your browser does not support the audio element.

Now, define a function for displaying a spectrogram:

Plot the example's waveform over time and the corresponding spectrogram (frequencies over time):

Now, create spectrogram datasets from the audio datasets:

Examine the spectrograms for different examples of the dataset:

Build and train the model

Add Dataset.cache and Dataset.prefetch operations to reduce read latency while training the model:

For the model, you'll use a simple convolutional neural network (CNN), since you have transformed the audio files into spectrogram images.

Your tf.keras.Sequential model will use the following Keras preprocessing layers:

• tf.keras.layers.Resizing : to downsample the input to enable the model to train faster.
• tf.keras.layers.Normalization : to normalize each pixel in the image based on its mean and standard deviation.

For the Normalization layer, its adapt method would first need to be called on the training data in order to compute aggregate statistics (that is, the mean and the standard deviation).

Configure the Keras model with the Adam optimizer and the cross-entropy loss:

Train the model over 10 epochs for demonstration purposes:

Let's plot the training and validation loss curves to check how your model has improved during training:

Evaluate the model performance

Run the model on the test set and check the model's performance:

Display a confusion matrix

Use a confusion matrix to check how well the model did classifying each of the commands in the test set:

Run inference on an audio file

Finally, verify the model's prediction output using an input audio file of someone saying "no". How well does your model perform?

As the output suggests, your model should have recognized the audio command as "no".

Export the model with preprocessing

The model's not very easy to use if you have to apply those preprocessing steps before passing data to the model for inference. So build an end-to-end version:

Test run the "export" model:

Save and reload the model, the reloaded model gives identical output:

This tutorial demonstrated how to carry out simple audio classification/automatic speech recognition using a convolutional neural network with TensorFlow and Python. To learn more, consider the following resources:

• The Sound classification with YAMNet tutorial shows how to use transfer learning for audio classification.
• The notebooks from Kaggle's TensorFlow speech recognition challenge .
• The TensorFlow.js - Audio recognition using transfer learning codelab teaches how to build your own interactive web app for audio classification.
• A tutorial on deep learning for music information retrieval (Choi et al., 2017) on arXiv.
• TensorFlow also has additional support for audio data preparation and augmentation to help with your own audio-based projects.
• Consider using the librosa library for music and audio analysis.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License , and code samples are licensed under the Apache 2.0 License . For details, see the Google Developers Site Policies . Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2024-08-16 UTC.

Automatic Speech Recognition using CTC

Authors: Mohamed Reda Bouadjenek and Ngoc Dung Huynh Date created: 2021/09/26 Last modified: 2021/09/26 Description: Training a CTC-based model for automatic speech recognition.

Introduction

Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken language into text by computers. It is also known as automatic speech recognition (ASR), computer speech recognition or speech to text (STT). It incorporates knowledge and research in the computer science, linguistics and computer engineering fields.

This demonstration shows how to combine a 2D CNN, RNN and a Connectionist Temporal Classification (CTC) loss to build an ASR. CTC is an algorithm used to train deep neural networks in speech recognition, handwriting recognition and other sequence problems. CTC is used when we don’t know how the input aligns with the output (how the characters in the transcript align to the audio). The model we create is similar to DeepSpeech2 .

We will use the LJSpeech dataset from the LibriVox project. It consists of short audio clips of a single speaker reading passages from 7 non-fiction books.

We will evaluate the quality of the model using Word Error Rate (WER) . WER is obtained by adding up the substitutions, insertions, and deletions that occur in a sequence of recognized words. Divide that number by the total number of words originally spoken. The result is the WER. To get the WER score you need to install the jiwer package. You can use the following command line:

References:

• LJSpeech Dataset
• Speech recognition
• Sequence Modeling With CTC
• DeepSpeech2

Load the LJSpeech Dataset

Let's download the LJSpeech Dataset . The dataset contains 13,100 audio files as wav files in the /wavs/ folder. The label (transcript) for each audio file is a string given in the metadata.csv file. The fields are:

• ID : this is the name of the corresponding .wav file
• Transcription : words spoken by the reader (UTF-8)
• Normalized transcription : transcription with numbers, ordinals, and monetary units expanded into full words (UTF-8).

For this demo we will use on the "Normalized transcription" field.

Each audio file is a single-channel 16-bit PCM WAV with a sample rate of 22,050 Hz.

We now split the data into training and validation set.

Preprocessing

We first prepare the vocabulary to be used.

Next, we create the function that describes the transformation that we apply to each element of our dataset.

Creating Dataset objects

We create a tf.data.Dataset object that yields the transformed elements, in the same order as they appeared in the input.

Visualize the data

Let's visualize an example in our dataset, including the audio clip, the spectrogram and the corresponding label.

Your browser does not support the audio element.

We first define the CTC Loss function.

We now define our model. We will define a model similar to DeepSpeech2 .

Training and Evaluating

Let's start the training process.

In practice, you should train for around 50 epochs or more. Each epoch takes approximately 5-6mn using a GeForce RTX 2080 Ti GPU. The model we trained at 50 epochs has a Word Error Rate (WER) ≈ 16% to 17% .

Some of the transcriptions around epoch 50:

Audio file: LJ017-0009.wav

Audio file: LJ003-0340.wav

Audio file: LJ011-0136.wav

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

• Stack Overflow for Teams Where developers & technologists share private knowledge with coworkers
• Advertising & Talent Reach devs & technologists worldwide about your product, service or employer brand
• OverflowAI GenAI features for Teams
• OverflowAPI Train & fine-tune LLMs
• Labs The future of collective knowledge sharing
• About the company Visit the blog

Collectives™ on Stack Overflow

Find centralized, trusted content and collaborate around the technologies you use most.

Q&A for work

Connect and share knowledge within a single location that is structured and easy to search.

Get early access and see previews of new features.

How to train CNN on common voice dataset

I am trying to train a cnn with the common voice dataset. I am new to speech recognition and am not able to find any links on how to use the dataset with keras. I followed this article to build a simple word classification network. But I want to scale it up with the common voice dataset. any help is appreciated.

• conv-neural-network
• speech-recognition

• 1 What is the end goal that you want to achieve? Speech recognition? or what are your labels? –  Edward Aung Commented Aug 1, 2019 at 5:18
• My end goal is speech to text conversion. –  Sashaank Commented Aug 1, 2019 at 5:19
• 1 The server for the blog article you linked to seems to be down. That makes it impossible to comment in a meaningful way. I'd like to suggest "smaller", answerable questions about concrete problems instead of "how do I do <large problem>". –  Hendrik Commented Aug 1, 2019 at 6:44
• sorry about the link. for some reason the link opens properly through the medium app in android, but fails to open though the browser. –  Sashaank Commented Aug 1, 2019 at 9:00

What you can do is looking at MFCCs . In short, these are features extracted from the audio waveform by using signal processing techniques to transcribe the way humans perceive sound. In python, you can use python-speech-features to compute MFCCs.

Once you have prepared your data, you can build a CNN; for example something like this one :

You can also use RNNs (LSTM or GRU for example), but this is a bit more advanced.

EDIT: A very good dataset to start, if you want:

Speech Commands Dataset

• Thanks for the reply. I will certainly do that. Just a follow on doubt I have is, the common voice dataset is a compilation of sentences spoken by various people. For speech recognition should I convert there sentences to words? –  Sashaank Commented Aug 1, 2019 at 8:58
• It's much easier to work with labeled words than with sentences; you can work with sentences using RNN + CTC-loss for example, but its very advanced. You may practice with words before! If you want a dataset with already prepared words, you can take a look to the google speech commands dataset (i'll put the link in my answer). This is a very good dataset to start. –  Baptiste Pouthier Commented Aug 1, 2019 at 9:05

Your Answer

Reminder: Answers generated by artificial intelligence tools are not allowed on Stack Overflow. Learn more

Sign up or log in

Post as a guest.

Required, but never shown

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy .

Not the answer you're looking for? Browse other questions tagged python keras conv-neural-network speech-recognition librosa or ask your own question .

• The Overflow Blog
• From PHP to JavaScript to Kubernetes: how one backend engineer evolved over time
• Featured on Meta
• We've made changes to our Terms of Service & Privacy Policy - July 2024
• Bringing clarity to status tag usage on meta sites
• Feedback requested: How do you use tag hover descriptions for curating and do...
• What does a new user need in a homepage experience on Stack Overflow?

Hot Network Questions

• Does gluing two points prevent simple connectedness?
• Whether and when this sum will converge?
• How to raise a vector to powers contained in a vector, change the list into a product, and do this for all the lines of a matrix, efficiently?
• How should I respond to a former student from my old institution asking for a reference?
• When was this photo taken?
• How to justify a ban on exclusivist religions?
• How did the cop infer from Uncle Aaron's statement that Miles has been visiting?
• Will this be the first time, that there are more People an ISS than seats in docked Spacecraft?
• Are chord inversions determined solely by the lowest note, even in a broken chord?
• If physics can be reduced to mathematics (and thus to logic), does this mean that (physical) causation is ultimately reducible to implication?
• Can the subjunctive mood be combined with ‘Be to+infinitive’?
• Is there anything that stops the majority shareholder(s) from destroying company value?
• Millennial reign and New Heaven and New Earth
• How can one says that a particle IS a representation of some group?
• Are there any virtues in virtue ethics that cannot be plausibly grounded in more fundamental utilitarian principles?
• What do the hexadecimal numbers represent on the ls output of the /dev directory?
• If Miles doesn’t consider Peter’s actions as hacking, then what does he think Peter is doing to the computer?
• What does 北渐 mean?
• Why if gravity were higher, designing a fully reusable rocket would be impossible?
• How does one go about writing papers as a nobody?
• When testing for normally distributed data, should I consider all variables before running shapiro.test?
• Can I use rear (thru) axle with crack for a few rides, before getting a new one?
• Incident in WWII re US troops in non-segregated British pubs
• In the US, can I buy iPhone and Android phones and claim them as expense?

• Skip to main content
• Keyboard shortcuts for audio player

2024 Election

Fact checking trump's claims during harris' acceptance speech.

Former President Donald Trump speaks at the U.S.-Mexico border Thursday near Sierra Vista, Ariz. Rebecca Noble/Getty Images hide caption

Former President Donald Trump told his followers on Truth Social on Wednesday that he would be posting throughout Kamala Harris' DNC speech, when she formally accepted the party's nomination for president.

Here are some of the issues Trump commented on while Harris spoke, with some quick fact-checking.

On abortion: "Everybody, Democrats, Republicans, Liberals, and Conservatives, wanted Roe v. Wade TERMINATED , and brought back to the States."

False:  According to a Gallup poll from June 2023 , one year after Roe v. Wade was overturned, 61% of respondents said overturning Roe  was a "bad thing," while 38% said it was a "good thing."

Additionally, an NPR/PBS NewsHour/Marist poll  from earlier this year showed that most Americans believe criminalizing abortion is wrong.

162 lies and distortions in a news conference. NPR fact-checks former President Trump

On immigration:  "She just called to give all Illegals CITIZENSHIP , SAY GOODBYE TO THE U.S.A.! SHE IS A RADICAL MARXIST!"

False:  During her acceptance speech tonight, Harris said she would support a bipartisan border bill on immigration. There is nothing in the text of the bill that would give all undocumented immigrants automatic American citizenship.

Additionally, while Harris mentioned that pathways to citizenship should exist, this does not equate to automatic citizenship for those in the country illegally.

On his legal troubles:  " These Prosecutions were all started by her and Biden against their Political Opponent, ME!"

False: The White House has nothing to do with the cases brought against former President Trump, whose four current criminal cases were brought against him by the New York state court, the U.S. District Court for the District of Columbia, the Georgia state court and the U.S. District Court for the Southern District of Florida, respectively.

• election 2024
• From Live Coverage
• Donald Trump

Advertisement

Full Transcript of Kamala Harris’s Democratic Convention Speech

The vice president’s remarks lasted roughly 35 minutes on the final night of the convention in Chicago.

• Share full article

By The New York Times

• Aug. 23, 2024

This is a transcript of Vice President Kamala Harris’s speech on Thursday night in which she formally accepted the Democratic Party’s nomination for the presidency.

OK, let’s get to business. Let’s get to business. All right.

So, let me start by thanking my most incredible husband, Doug. For being an incredible partner to me, an incredible father to Cole and Ella, and happy anniversary, Dougie. I love you so very much.

To our president, Joe Biden. When I think about the path that we have traveled together, Joe, I am filled with gratitude. Your record is extraordinary, as history will show, and your character is inspiring. And Doug and I love you and Jill, and are forever thankful to you both.

And to Coach Tim Walz. You are going to be an incredible vice president. And to the delegates and everyone who has put your faith in our campaign, your support is humbling.

So, America, the path that led me here in recent weeks was, no doubt, unexpected. But I’m no stranger to unlikely journeys. So, my mother, our mother, Shyamala Harris, had one of her own. And I miss her every day, and especially right now. And I know she’s looking down smiling. I know that.

So, my mother was 19 when she crossed the world alone, traveling from India to California with an unshakable dream to be the scientist who would cure breast cancer.

We are having trouble retrieving the article content.

Please enable JavaScript in your browser settings.

Thank you for your patience while we verify access. If you are in Reader mode please exit and  log into  your Times account, or  subscribe  for all of The Times.

Thank you for your patience while we verify access.

Already a subscriber?  Log in .

Want all of The Times?  Subscribe .

Navigation Menu

Search code, repositories, users, issues, pull requests..., provide feedback.

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly.

To see all available qualifiers, see our documentation .

sign-language-recognition-system

Here are 77 public repositories matching this topic..., harshbg / sign-language-interpreter-using-deep-learning.

A sign language interpreter using live video feed from the camera.

• Updated Apr 18, 2024

hthuwal / sign-language-gesture-recognition

Sign Language Gesture Recognition From Video Sequences Using RNN And CNN

• Updated Sep 27, 2020

loicmarie / sign-language-alphabet-recognizer

Simple sign language alphabet recognizer using Python, openCV and tensorflow for training Inception model (CNN classifier).

• Updated Jul 9, 2023

jackyjsy / CVPR21Chal-SLR

This repo contains the official code of our work SAM-SLR which won the CVPR 2021 Challenge on Large Scale Signer Independent Isolated Sign Language Recognition.

• Updated Nov 16, 2022

0aqz0 / SLR

isolated & continuous sign language recognition using CNN+LSTM/3D CNN/GCN/Encoder-Decoder

• Updated May 12, 2020

Arshad221b / Sign-Language-Recognition

Indian Sign language Recognition using OpenCV

• Updated Jun 12, 2023
• Jupyter Notebook

Tachionstrahl / SignLanguageRecognition

Real-time Recognition of german sign language (DGS) with MediaPipe

• Updated Oct 9, 2022

rrupeshh / Simple-Sign-Language-Detector

Simple Sign Language Detector

• Updated Jul 12, 2018

soumik12345 / Kinect-Vision

A computer vision based gesture detection system that automatically detects the number of fingers as a hand gesture and enables you to control simple button pressing games using you hand gestures.

• Updated May 3, 2020

Mquinn960 / sign-language

Android application which uses feature extraction algorithms and machine learning (SVM) to recognise and translate static sign language gestures.

• Updated May 11, 2019

jayshah19949596 / DeepSign-A-Deep-Learning-Architecture-for-Sign-Language-Recognition

• Updated Sep 30, 2019

MaheshNat / Signify

A simple sign language detection web app built using Next.js and Tensorflow.js. 2020 Congressional App Challenge. Winner! Developed by Mahesh Natamai and Arjun Vikram.

• Updated May 9, 2021

MuhammadMoinFaisal / Sign-Language-Alphabets-Detection-and-Recongition-using-YOLOv8

Sign Language Alphabet Detection and Recognition using YOLOv8

• Updated Jan 19, 2023

surdoparasurdo / awesome-sign-language

🙌 A collection of awesome Sign Language projects and resources 🤟

• Updated Feb 20, 2019

Elysian01 / Sign-Language-Translator

Sign Language Translator enables the hearing impaired user to communicate efficiently in sign language, and the application will translate the same into text/speech. The user has to train the model, by recording its own sign language gestures. Internally it uses MobileNet and KNN classifier to classify the gestures.

• Updated Apr 9, 2024

jackyjsy / SAM-SLR-v2

SAM-SLR-v2 is an improved version of SAM-SLR for sign language recognition.

• Updated Oct 20, 2021

209sontung / sign-language

Real-time Sign Language Gesture Recognition Using 1DCNN + Transformers on MediaPipe landmarks

• Updated Oct 5, 2023

baidwan007 / Sign-Languge-to-speech-conversion

We help the deaf and the dumb to communicate with normal people using hand gesture to speech conversion. In this code we use depth maps from the kinect camera and techniques like convex hull + contour mapping to recognise 5 hand signs

• Updated Jul 27, 2017

ai-forever / easy_sign

Easy_sign is an open source russian sign language recognition project that uses small CPU model for predictions and is designed for easy deployment via Streamlit.

• Updated Dec 28, 2023

shubhammore1251 / Sign-Language-Recognition-Using-Mediapipe-and-React

A Sign Language Learning Platform where who know sign language can come and practice Sign Language and also people who don't know can learn through this

• Updated Aug 19, 2024

Improve this page

Add a description, image, and links to the sign-language-recognition-system topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the sign-language-recognition-system topic, visit your repo's landing page and select "manage topics."