Predição de chuva DIY usando Arduino, Python e Keras

Etapa 4:Preparando a estação base

As etapas de instalação:

  mongouse weather db.weather_station.createIndex ({"date":1})  

  sudo apt-get install clone gitgit https://github.com/danionescu0/home-automation.gi ...  

  cd weather-predict # cria um ambiente anaconda chamado "weather" com python 3.6.2conda create --name weather python =3.6.2 # activate environmentconda activate weather # instala todos os pacotes pip install -r requirements.txt  

Configure o projeto

  mongodb ={'host':'localhost', 'port':27017} ...  

  ls -l / dev / tty *  

  serial ={'port':'/ dev / ttyUSB0', 'baud_rate':9600}  

Etapa 5:use a estação meteorológica na prática

  mongoimport -d clima -c weather_station --file sample_data / weather_station.json  

  conda ativar weatherpython serial_listener.py # a cada 10 minutos você deve ver os dados da estação meteorológica chegando:[Sensor:tipo (temperatura), valor (14.3)] [Sensor:tipo (pressão), valor ( 1056.0)] ...  

  conda activate weatherpython train.py --days_behind 600 --test-file-percent 10 --datapoints-behind 8 --hour-granularity 6  

  conda ativar gráficos de clima python - dias atrás 10  

  conda ativar weather python predict.py --datapoints-behind 8 --hour-granularity 6 --from-addr a_gmail_address --from-password gmail_password --to-addr a_email_destination  

  python predict_batch.py ​​-f sample_data / test_data.csv  

Etapa 6:como funciona

Nesta última etapa, discutiremos vários aspectos da arquitetura deste projeto:

1. Visão geral do projeto, discutiremos a arquitetura geral e as tecnologias envolvidas

• requisitos de energia, se a estação meteorológica também pudesse processar os dados, ela precisaria de energia substancial, talvez grandes painéis solares ou uma fonte de energia permanente

• portabilidade, devido ao seu tamanho pequeno, a estação meteorológica pode coletar dados a algumas centenas de metros de distância e você pode facilmente mudar de lugar se necessário

• escalabilidade, você pode aumentar a precisão da previsão construindo mais de uma estação meteorológica e espalhando-as por algumas centenas de metros

• baixo custo, porque é um dispositivo barato, você pode facilmente construir outro no caso de um ser perdido ou roubado

  {"_id":"04_27_2017_06_17", "umidade":65, "data":ISODate ("2017-04-27T06:17:18Z"), "pressão":1007, "temperatura":9, "chuva":0, "luz":15}  

2. Conceitos básicos de aprendizado de máquina

• estimular os neurônios de entrada

• propagar os sinais através das camadas da rede até os neurônios de saída

• leia os neurônios de saída e compare os resultados com os resultados desejados

• ajuste os pesos dos axônios para um melhor resultado da próxima vez

• comece novamente até que o número de loops seja alcançado

3. Como os dados são preparados (a etapa mais importante)

• obtenha os dados brutos

• limpeza de dados:isso significará remover valores órfãos, aberrações ou outras anomalias

• agrupamento de dados:pegando muitos pontos de dados e transformando em um ponto de dados agregado

• aprimoramento de dados:adicionar outros aspectos dos dados derivados de seus próprios dados ou de fontes externas

• divisão dos dados em treinamento e dados de teste

• divida cada um dos dados de trem e teste em entradas e saídas. Normalmente, um problema terá muitas entradas e poucas saídas

• rescale the data so it's between 0 and 1 (this will help the network removing high/low value biases)

client =MongoClient(host, port).weather.datapoints cursor =client.find( {'$and' :[ {'date' :{'$gte' :start_date}}, {'date' :{'$lte' :end_date}} ]} )data =list(cursor).. 

dataframe =dataframe.dropna() 

• min value

• max value

• mean

• 70, 90, 30, 10 percentiles

• nr of times there has been a rise in a sensor

• nr of times there has been a fall in a sensor

• nr of times there has been steady values in a sensor

_id date humidity light pressure rain temperature 04_27_2017_03_08 2017-04-27 03:08:36 67.0 0.0 1007.0 0.0 11.004_27_2017_03_19 2017-04-27 03:19:05 66.0 0.0 1007.0 0.0 11.004_27_2017_03_29 2017-04-27 03:29:34 66.0 0.0 1007.0 0.0 11.0  

_id date humidity_10percentile humidity_30percentile humidity_70percentile humidity_90percentile humidity_avg ... temperature_avg temperature_fall temperature_max temperature_min temperature_rise temperature_steady ... 04_27_2017_0 2017-04-27 03:08:36 59.6 60.8 63.2 66.0 62.294118 ... 10.058824 2 11.0 9.0 1 1404_27_2017_1 2017-04-27 06:06:50 40.3 42.0 60.0 62.0 50.735294 ... 14.647059 3 26.0 9.0 11 2004_27_2017_2 2017-04-27 12:00:59 36.0 37.0 39.8 42.0 38.314286 ... 22.114286 1 24.0 20.0 5 29  

dataframe.insert(loc=1, column='has_rain', value=numpy.where(dataframe['rain_avg']> 0.1, 1, 0)) 

dataframe =dataframe.drop(['date'], axis=1)dataframe =dataframe[dataframe['temperature_min']>=0] 

_id has_rain humidity_10percentile humidity_30percentile humidity_70percentile humidity_90percentile ... temperature_steady_4 temperature_steady_5 temperature_steady_6 temperature_steady_7 temperature_steady_8 ... 04_27_2017_3 0 36.0 44.8 61.0 63.0 ... NaN NaN NaN NaN NaN04_28_2017_0 0 68.0 70.0 74.0 75.0 ... 14.0 NaN NaN NaN NaN04_28_2017_1 0 40.0 45.0 63.2 69.0 ... 20.0 14.0 NaN NaN NaN04_28_2017_2 0 34.0 35.9 40.0 41.0 ... 29.0 20.0 14.0 NaN NaN04_28_2017_3 0 36.1 40.6 52.0 54.0 ... 19.0 29.0 20.0 14.0 NaN04_29_2017_0 0 52.0 54.0 56.0 58.0 ... 26.0 19.0 29.0 20.0 14.004_29_2017_1 0 39.4 43.2 54.6 57.0 ... 18.0 26.0 19.0 29.0 20.004_29_2017_2 1 41.0 42.0 44.2 47.0 ... 28.0 18.0 26.0 19.0 29.0  

from sklearn.model_selection import train_test_split ...main_data, test_data =train_test_split(dataframe, test_size=percent_test_data) ... 

X =main_data.iloc[:, 1:].valuesy =main_data.iloc[:, 0].values  

from sklearn.preprocessing import StandardScalerfrom sklearn.externals import joblib..scaler =StandardScaler()X =scaler.fit_transform(X) ...# of course we should be careful to save the scaled model for later reusejoblib.dump(scaler, 'model_file_name.save')  

4. How the actual neural network wrapper API works (Keras)

from keras.models import Sequentialfrom keras.layers import Densefrom keras.layers import Dropout ...input_dimensions =X.shape[1] optimizer ='rmsprop'dropout =0.05model =Sequential()inner_nodes =int(input_dimensions / 2)model.add(Dense(inner_nodes, kernel_initializer='uniform', activation='relu', input_dim=input_dimensions))model.add(Dropout(rate=dropout))model.add(Dense(inner_nodes, kernel_initializer='uniform', activation='relu'))model.add(Dropout(rate=dropout))model.add(Dense(1, kernel_initializer='uniform', activation='sigmoid'))model.compile(optimizer=optimizer, loss='mean_absolute_error', metrics=['accuracy']) model.fit(X, y, batch_size=1, epochs=50)...# save the model for later useclassifier.save('file_model_name') 

• activation functions (sigmoid, relu). This are functions that dictate when the neuron will transmit it's impulse further in the network. There are many, but sigmoid and relu are the most common. Check out this link for more details:https://towardsdatascience.com/activation-function...

• kernel_initializer function (uniform). This means that all the weights are initialized with random uniform values

• loss function (mean_absolute_error). This function measures the error comparing the network predicted result versus the ground truth. There are many alternatives:https://keras.io/losses/

• metrics function (accuracy). It measures the performance of the model

• optimiser functions (rmsprop). It optimizes how the model learn through backpropagation.

• batch_size. Number of datapoints to take once by Keras before applying optimizer function

• epochs:how many times the process it's started from 0 (to learn better)

5. Future improvements

• add a wind speed / direction sensor. This could be a very important sensor that i'm missing in my model

• experiment with UV rays, gas and particle sensors

• add at least two stations in the zone for better data (make some better averages)

• collect a few more years of data, i've experimented with just a year and a half

• try to incorporate data from other sources into the model. You can start to import wind speed data and combine with the local station data for a better model. This website offers historical data:https://www.wunderground.com/history/

• optimize the Keras model better by adjusting:layers, nr of neurons in layers, dropout percents, metrics functions, optimiser functions, loss functions, batch size, learning epochs

• try other model architectures, for example i've experimented with LSTM (long short term memory) but it gived slightly poorer results)

python train.py --days_behind 600 --test-file-percent 10 --datapoints-behind 6 --hour-granularity 6 --grid-search 

Step 7:Bonus:Using an Official Weather Dataset

• temperature

• humidity

• pressure

• wind speed

• wind gust

• ub index

• visibilitySo this beeing data from an official weather station, and having more parameters I thought it should perform better so i've gave it a try. To replicate my findings:

• to download your self, first create an account in darsky and get the API key

• replace the API key in download_import/config.py

• also in the config replace the geographic coordonates for the location you want to predict the rain

• in a console activate "weather" anaconda environment and run:

python download_import/darksky.py -d 1000 

mongoimport -d weather -c darksky --file sample_data/darksky.json  

python train.py -d 2000 -p 20 -dp 4 -hg 6 --data-source darksky 

python predict_batch.py -f sample_data/test_data.csv