Indices Corruptos en Graylog: Deflector Exists as an Index and Is Not an Alias

Problema: graylog deja de funcionar y los nodos del cluster no van: estan en rojo y tus busquedas no funcionan….algo pasa :)

Te encuentras con estos logs en graylog

1
2
3
4
5
6
7
Caused by: java.lang.NumberFormatException: For input string: "INFO"
    at java.lang.NumberFormatException.forInputString(Unknown Source) ~[?:1.8.0_131]
    at java.lang.Long.parseLong(Unknown Source) ~[?:1.8.0_131]
    at java.lang.Long.parseLong(Unknown Source) ~[?:1.8.0_131]
12:05
org.elasticsearch.index.mapper.MapperParsingException: failed to parse [level]

Sacas un listado de los indices y hay algunos en rojo como graylog_deflector

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<<< DEV >>> root@elasticsearch-3:/home/me# curl -XGET localhost:9200/_cat/indices/
green  open infra_256          kGaMkaRFSC6U9Jmwy12rcg 1 0   865668    0 952.9mb 952.9mb
yellow open admin              SKOJKDG9Su22Gaky88EYFg 5 1        0    0    955b    955b
green  open graylog_1330       Dfg0gxIUQZWKk4befHPDgg 1 0  3246209    0   984mb   984mb
green  open infra_268          hBWgkwWoQVq56sLGoutiZw 1 0   742742    0 934.6mb 934.6mb
green  open graylog_1343       3hNgjtbMSo2vjLBFe9EwuQ 1 0  2718275    0 905.9mb 905.9mb
green  open graylog_1341       prnmoaZ4TqWZvk5TwAku8w 1 0  2453011    0   770mb   770mb
green  open infra_251          KamoFJo-S2S7LIIHcN2flw 1 0   739868    0 948.4mb 948.4mb
green  open infra_248          ge77cCZBSZuTwmq0wr-Mcg 1 0   574679    0 747.8mb 747.8mb
green  open graylog_1342       BzQtkgOkSf6Scp0ZPiqGfA 1 0  2446182    0 792.1mb 792.1mb
green  open infra_263          yP1kpJFATS2uuh1v1mOR8Q 1 0   655589    0 927.1mb 927.1mb
green  open infra_259          i1s6JlzNRTOmEB2Kq2GzTg 1 0   624826    0 852.1mb 852.1mb
green  open snmp_0             _1Nqdwz_SxmObUSnNNdJgA 1 0        0    0    191b    191b
green  open infra_241          OL0B4RJjRWyyEilxb51jcQ 1 0   806433    0   939mb   939mb
green  open infra_254          0ru9BT5pTYeNYV4uAYC_cQ 1 0   628605    0   895mb   895mb
green  open infra_240          Q_xMSj1tRgy5Iudm5i-jlw 1 0   809087    0 961.8mb 961.8mb
green  open infra_255          q26q4o_YQQe5l6ISfXH2qQ 1 0   590433    0 699.6mb 699.6mb
green  open infra_257          2oCAKjNnRKO4FyN-6qGcQA 1 0   618289    0 814.7mb 814.7mb
green  open graylog_1339       QNbCiY5YSMSmwhIMEPeeug 1 0  1925438    0 598.2mb 598.2mb
green  open infra_261          m2Jq0jpeQWybKw-4hoOHhg 1 0   567153    0 758.8mb 758.8mb
red    open graylog_deflector  IiS44uBOSYyBW5OLD5PmzA 5 1 33009570  799  13.3gb  13.3gb
green  open graylog_1340       bea8-Bb4RHqPmEhub9hAcQ 1 0  2366439    0 732.5mb 732.5mb
green  open infra_262          x-H5TBBlRDOn3d115ynlNg 1 0   706197    0 905.6mb 905.6mb

Y el graylog te dice que Deflector exists as an index and is not an alias

Solucion

Saca el estado de los indices, si hay en rojo: tienes posibles culpables

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<<< DEV >>> root@elasticsearch-3:/home/me# curl -XGET localhost:9200/_cat/indices/
green  open infra_256          kGaMkaRFSC6U9Jmwy12rcg 1 0   865668    0 952.9mb 952.9mb
yellow open admin              SKOJKDG9Su22Gaky88EYFg 5 1        0    0    955b    955b
green  open graylog_1330       Dfg0gxIUQZWKk4befHPDgg 1 0  3246209    0   984mb   984mb
green  open infra_268          hBWgkwWoQVq56sLGoutiZw 1 0   742742    0 934.6mb 934.6mb
green  open graylog_1343       3hNgjtbMSo2vjLBFe9EwuQ 1 0  2718275    0 905.9mb 905.9mb
green  open graylog_1341       prnmoaZ4TqWZvk5TwAku8w 1 0  2453011    0   770mb   770mb
green  open infra_251          KamoFJo-S2S7LIIHcN2flw 1 0   739868    0 948.4mb 948.4mb
green  open infra_248          ge77cCZBSZuTwmq0wr-Mcg 1 0   574679    0 747.8mb 747.8mb
green  open graylog_1342       BzQtkgOkSf6Scp0ZPiqGfA 1 0  2446182    0 792.1mb 792.1mb
green  open infra_263          yP1kpJFATS2uuh1v1mOR8Q 1 0   655589    0 927.1mb 927.1mb
green  open infra_259          i1s6JlzNRTOmEB2Kq2GzTg 1 0   624826    0 852.1mb 852.1mb
green  open snmp_0             _1Nqdwz_SxmObUSnNNdJgA 1 0        0    0    191b    191b
green  open infra_241          OL0B4RJjRWyyEilxb51jcQ 1 0   806433    0   939mb   939mb
green  open infra_254          0ru9BT5pTYeNYV4uAYC_cQ 1 0   628605    0   895mb   895mb
green  open infra_240          Q_xMSj1tRgy5Iudm5i-jlw 1 0   809087    0 961.8mb 961.8mb
green  open infra_255          q26q4o_YQQe5l6ISfXH2qQ 1 0   590433    0 699.6mb 699.6mb
green  open infra_257          2oCAKjNnRKO4FyN-6qGcQA 1 0   618289    0 814.7mb 814.7mb
green  open graylog_1339       QNbCiY5YSMSmwhIMEPeeug 1 0  1925438    0 598.2mb 598.2mb
green  open infra_261          m2Jq0jpeQWybKw-4hoOHhg 1 0   567153    0 758.8mb 758.8mb
red    open graylog_deflector  IiS44uBOSYyBW5OLD5PmzA 5 1 33009570  799  13.3gb  13.3gb
green  open graylog_1340       bea8-Bb4RHqPmEhub9hAcQ 1 0  2366439    0 732.5mb 732.5mb
green  open infra_262          x-H5TBBlRDOn3d115ynlNg 1 0   706197    0 905.6mb 905.6mb

Saca el listado de los que tengas en estado UNASSIGNED

1
curl -s -XGET 10.25.152.30:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | sort | less

Mira a ver cuando datos tienes esos que te aparecen en rojo / UNASSIGNED
Para el Graylog
Elimina los indices corruptos

1
curl -X DELETE http://localhost:9200/[indices-podridos]

Haz otro cat de los indices

1
curl -X DELETE http://localhost:9200/.triggered_watches

Arranca graylog
Funciona? Enhorabuena!

Comments

18-08-2018
⬆︎TOP