Archive-name: ai-faq/neural-nets/part4 Last-modified: 1999-11-21 URL: ftp://ftp.sas.com/pub/neural/FAQ4.html Maintainer: saswss@unx.sas.com (Warren S. Sarle)
This is part 4 (of 7) of a monthly posting to the Usenet newsgroup comp.ai.neural-nets. See the part 1 of this posting for full information what it is all about.
------------------------------------------------------------------------
Barnes & Noble http://www.bn.com/ seems to have a slightly larger stock of technical books than Amazon, but their search engine is inferior and their web response is slower. It's a good idea to check both Barnes & Noble and Amazon for availability and prices.
Bookpool at http://www.bookpool.com/ does not have as large a selection as Amazon or Barnes and Noble but they often offer exceptional discounts.
The neural networks reading group at the University of Illinois at Urbana-Champaign, the Artifical Neural Networks and Computational Brain Theory (ANNCBT) forum, has compiled a large number of book and paper reviews at http://anncbt.ai.uiuc.edu/, with an emphasis more on cognitive science rather than practical applications of NNs.
If you have questions on feedforward nets that aren't answered by Bishop, try Masters (1993) or Reed and Marks (1999) for practical issues or Ripley (1996) for theortical issues, all of which are reviewed below.
Bigus has written an excellent introduction to NNs for the SBE. Bigus says (p. xv), "For business executives, managers, or computer professionals, this book provides a thorough introduction to neural network technology and the issues related to its application without getting bogged down in complex math or needless details. The reader will be able to identify common business problems that are amenable to the neural netwrk approach and will be sensitized to the issues that can affect successful completion of such applications." Bigus succeeds in explaining NNs at a practical, intuitive, and necessarily shallow level without formulas--just what the SBE needs. This book is far better than Caudill and Butler (1990), a popular but disastrous attempt to explain NNs without formulas.
Chapter 1 introduces data mining and data warehousing, and sketches some applications thereof. Chapter 2 is the semi-obligatory philosophico-historical discussion of AI and NNs and is well-written, although the SBE in a hurry may want to skip it. Chapter 3 is a very useful discussion of data preparation. Chapter 4 describes a variety of NNs and what they are good for. Chapter 5 goes into practical issues of training and testing NNs. Chapters 6 and 7 explain how to use the results from NNs. Chapter 8 discusses intelligent agents. Chapters 9 through 12 contain case histories of NN applications, including market segmentation, real-estate pricing, customer ranking, and sales forecasting.
Bigus provides generally sound advice. He briefly discusses overfitting and overtraining without going into much detail, although I think his advice on p. 57 to have at least two training cases for each connection is somewhat lenient, even for noise-free data. I do not understand his claim on pp. 73 and 170 that RBF networks have advantages over backprop networks for nonstationary inputs--perhaps he is using the word "nonstationary" in a sense different from the statistical meaning of the term. There are other things in the book that I would quibble with, but I did not find any of the flagrant errors that are common in other books on NN applications such as Swingler (1996).
The one serious drawback of this book is that it is more than one page long and may therefore tax the attention span of the SBE. But any SBE who succeeds in reading the entire book should learn enough to be able to hire a good NN expert to do the real work.
Weiss, S.M. and Kulikowski, C.A. (1991), Computer Systems That
Learn, Morgan Kaufmann. ISBN 1 55860 065 5.
Author's Webpage: Kulikowski: http://ruccs.rutgers.edu/faculty/kulikowski.html
Book Webpage (Publisher): http://www.mkp.com/books_catalog/1-55860-065-5.asp
Additional Information: Information of Weiss, S.M. are not available.
Briefly covers at a very elementary level feedforward nets, linear and
nearest-neighbor discriminant analysis, trees, and expert sytems,
emphasizing practical applications. For a book at this level, it has an
unusually good chapter on estimating generalization error, including
bootstrapping.
Reed, R.D., and Marks, R.J, II (1999),
Neural Smithing: Supervised Learning in Feedforward Artificial
Neural Networks, Cambridge, MA: The MIT Press, ISBN 0-262-18190-8.
Author's Webpage: Marks: http://cialab.ee.washington.edu/Marks.html
Book Webpage (Publisher):
http://mitpress.mit.edu/book-home.tcl?isbn=0262181908
After you have read Smith (1993) or Weiss and Kulikowski (1991), consult
Reed and Marks for practical details on training MLPs (other types of
neural nets such as RBF networks are barely even mentioned). They
provide extensive coverage of backprop and its variants, and they also
survey conventional optimization algorithms. Their coverage of
initialization methods, constructive networks, pruning, and
regularization methods is unusually thorough. Unlike the vast majority
of books on neural nets, this one has lots of really informative graphs.
The chapter on generalization assessment is slightly weak, which is why
you should read Smith (1993) or Weiss and Kulikowski (1991) first. Also,
there is little information on data preparation, for which Smith (1993)
and Masters (1993; see below) should be consulted. There is some
elementary calculus, but not enough that it should scare off anybody.
Many second-rate books treat neural nets as mysterious black boxes, but
Reed and Marks open up the box and provide genuine insight into the way
neural nets work.
One problem with the book is that the terms "validation set" and "test set" are used inconsistently.
Masters, T. (1995) Advanced Algorithms for Neural Networks:
A C++ Sourcebook, NY: John Wiley and Sons, ISBN 0-471-10588-0
Book Webpage (Publisher): http://www.wiley.com/
Additional Information: One has to search.
Clear explanations of conjugate gradient and Levenberg-Marquardt
optimization algorithms, simulated annealing, kernel regression (GRNN)
and discriminant analysis (PNN), Gram-Charlier networks, dimensionality
reduction, cross-validation, and bootstrapping.
Book Webpage (Publisher):
http://www.prenhall.com/books/esm_0133341860.html
Additional Information: The mentioned programs / additional support is not
available.
Review by Ian Cresswell:
What a relief! As a broad introductory text this is without any doubt the best currently available in its area. It doesn't include source code of any kind (normally this is badly written and compiler specific). The algorithms for many different kinds of simple neural nets are presented in a clear step by step manner in plain English.Anderson, J.A. (1995), An Introduction to Neural Networks, Cambridge,MA: The MIT Press, ISBN 0-262-01144-1.Equally, the mathematics is introduced in a relatively gentle manner. There are no unnecessary complications or diversions from the main theme.
The examples that are used to demonstrate the various algorithms are detailed but (perhaps necessarily) simple.
There are bad things that can be said about most books. There are only a small number of minor criticisms that can be made about this one. More space should have been given to backprop and its variants because of the practical importance of such methods. And while the author discusses early stopping in one paragraph, the treatment of generalization is skimpy compared to the books by Weiss and Kulikowski or Smith listed above.
If you're new to neural nets and you don't want to be swamped by bogus ideas, huge amounts of intimidating looking mathematics, a programming language that you don't know etc. etc. then this is the book for you.
In summary, this is the best starting point for the outsider and/or beginner... a truly excellent text.
Hertz, J., Krogh, A., and Palmer, R. (1991). Introduction to the Theory of
Neural Computation. Redwood City, CA: Addison-Wesley,
ISBN 0-201-50395-6 (hardbound) and 0-201-51560-1 (paperbound)
Book Webpage (Publisher):
http://www2.awl.com/gb/abp/sfi/computer.html
This is an excellent classic work on neural nets from the perspective of
physics. Comments from readers of comp.ai.neural-nets: "My first
impression is that this one is by far the best book on the topic. And
it's below $30 for the paperback."; "Well written, theoretical (but not
overwhelming)"; It provides a good balance of model development,
computational algorithms, and applications. The mathematical derivations
are especially well done"; "Nice mathematical analysis on the mechanism
of different learning algorithms"; "It is NOT for mathematical beginner.
If you don't have a good grasp of higher level math, this book can be
really tough to get through."
Devroye, L., Gy\"orfi, L., and Lugosi, G. (1996), A Probabilistic
Theory of Pattern Recognition, NY: Springer,
ISBN 0-387-94618-7, vii+636 pages.
Cichocki, A. and Unbehauen, R. (1993). Neural Networks for Optimization and
Signal Processing.
NY: John Wiley & Sons, ISBN 0-471-930105 (hardbound), 526 pages, $57.95.
Advanced:
Kosko, B. (1997), Fuzzy Engineering, Upper Saddle River,
NJ: Prentice Hall.
Beale, R. and Jackson, T. (1990). Neural Computing, an Introduction.
Adam Hilger, IOP Publishing Ltd : Bristol. (ISBN 0-85274-262-2).
Caudill, M. and Butler, C. (1990). Naturally Intelligent Systems.
MIT Press: Cambridge, Massachusetts. (ISBN 0-262-03156-6).
Chester, M. (1993). Neural Networks: A Tutorial,
Englewood Cliffs, NJ: PTR Prentice Hall.
Dayhoff, J. E. (1990). Neural Network Architectures: An Introduction.
Van Nostrand Reinhold: New York.
Freeman, James (1994). Simulating Neural Networks with
Mathematica, Addison-Wesley, ISBN: 0-201-56629-X.
Book Webpage (Publisher):
http://cseng.aw.com/bookdetail.qry?ISBN=0-201-56629-X&ptype=0
Freeman, J.A. and Skapura, D.M. (1991). Neural Networks:
Algorithms, Applications, and Programming Techniques,
Reading, MA: Addison-Wesley.
Gately, E. (1996). Neural Networks for Financial Forecasting.
New York: John Wiley and Sons, Inc.
Hecht-Nielsen, R. (1990). Neurocomputing. Addison Wesley.
McClelland, J. L. and Rumelhart, D. E. (1988).
Explorations in Parallel Distributed Processing: Computational Models of
Cognition and Perception (software manual). The MIT Press.
McCord Nelson, M. and Illingworth, W.T. (1990). A Practical Guide to Neural
Nets. Addison-Wesley Publishing Company, Inc. (ISBN 0-201-52376-0).
Muller, B., Reinhardt, J., Strickland, M. T. (1995). Neural
Networks.:An Introduction (2nd ed.). Berlin, Heidelberg, New York:
Springer-Verlag. ISBN 3-540-60207-0. (DOS 3.5" disk included.)
Orchard, G.A. & Phillips, W.A. (1991). Neural Computation: A
Beginner's Guide. Lawrence Earlbaum Associates: London.
Rao, V.B & H.V. (1993). C++ Neural Networks and Fuzzy Logic.
MIS:Press, ISBN 1-55828-298-x, US $45 incl. disks.
Wasserman, P. D. (1989). Neural Computing: Theory & Practice.
Van Nostrand Reinhold: New York. (ISBN 0-442-20743-3)
Rumelhart, D. E. and McClelland, J. L. (1986). Parallel Distributed
Processing: Explorations in the Microstructure of Cognition (volumes 1 & 2).
The MIT Press.
Knight, K. (1990). Connectionist, Ideas and Algorithms. Communications of
the ACM. November 1990. Vol.33 nr.11, pp 59-74.
Kohonen, T. (1988). An Introduction to Neural Computing. Neural Networks,
vol. 1, no. 1. pp. 3-16.
Rumelhart, D. E., Hinton, G. E. and Williams, R. J. (1986). Learning
representations by back-propagating errors. Nature, vol 323 (9 October),
pp. 533-536.
Anderson, J. A., Pellionisz, A. and Rosenfeld, E. (Eds). (1990).
Neurocomputing 2: Directions for Research. The MIT Press: Cambridge, MA.
Bourlard, H.A., and Morgan, N. (1994), Connectionist Speech
Recognition: A Hybrid Approach, Boston: Kluwer Academic
Publishers.
Deco, G. and Obradovic, D. (1996), An Information-Theoretic
Approach to Neural Computing, NY: Springer-Verlag.
Haykin, S. (1994). Neural Networks, a Comprehensive Foundation.
Macmillan, New York, NY.
Khanna, T. (1990). Foundations of Neural Networks. Addison-Wesley: New York.
Kung, S.Y. (1993). Digital Neural Networks, Prentice Hall,
Englewood Cliffs, NJ.
Book Webpage (Publisher):
http://www.prenhall.com/books/ptr_0136123260.html
Lippmann, R. P. (April 1987). An introduction to computing with neural nets.
IEEE Acoustics, Speech, and Signal Processing Magazine. vol. 2,
no. 4, pp 4-22.
Maren, A., Harston, C. and Pap, R., (1990). Handbook of Neural Computing
Applications. Academic Press. ISBN: 0-12-471260-6. (451 pages)
Pao, Y. H. (1989). Adaptive Pattern Recognition and Neural Networks
Addison-Wesley Publishing Company, Inc. (ISBN 0-201-12584-6)
Refenes, A. (Ed.) (1995). Neural Networks in the Capital Markets.
Chichester, England: John Wiley and Sons, Inc.
Simpson, P. K. (1990). Artificial Neural Systems: Foundations, Paradigms,
Applications and Implementations. Pergamon Press: New York.
Wasserman, P.D. (1993). Advanced Methods in Neural Computing.
Van Nostrand Reinhold: New York (ISBN: 0-442-00461-3).
Zeidenberg. M. (1990). Neural Networks in Artificial Intelligence.
Ellis Horwood, Ltd., Chichester.
Zornetzer, S. F., Davis, J. L. and Lau, C. (1990). An Introduction to
Neural and Electronic Networks. Academic Press. (ISBN 0-12-781881-2)
Zurada, Jacek M. (1992). Introduction To Artificial Neural Systems.
Hardcover, 785 Pages, 317 Figures, ISBN 0-534-95460-X, 1992, PWS Publishing
Company, Price: $56.75 (includes shipping, handling, and the ANS software
diskette). Solutions Manual available.
Mr Blum has not only contributed a masterpiece of NN inaccuracy
but also seems to lack a fundamental understanding of Object
Orientation.
The excessive use of virtual methods (see page 32 for
example), the inclusion of unnecessary 'friend' relationships (page
133) and a penchant for operator overloading (pick a page!)
demonstrate inability in C++ and/or OO.
The introduction to OO that is provided trivialises the area and
demonstrates a distinct lack of direction and/or understanding.
The public interfaces to classes are overspecified and the
design relies upon the flawed neuron/layer/network model.
There is a notable disregard for any notion of a robust class
hierarchy which is demonstrated by an almost total lack of concern
for inheritance and associated reuse strategies.
The attempt to rationalise differing types of Neural Network into
a single very shallow but wide class hierarchy is naive.
The general use of the 'float' data type would cause serious
hassle if this software could possibly be extended to use some of
the more sensitive variants of backprop on more difficult
problems. It is a matter of great fortune that such software is
unlikely to be reusable and will therefore, like all good
dinosaurs, disappear with the passage of time.
The irony is that there is a card in the back of the book asking
the unfortunate reader to part with a further $39.95 for a copy of
the software (already included in print) on a 5.25" disk.
The author claims that his work provides an 'Object Oriented
Framework ...'. This can best be put in his own terms (Page 137):
Before attempting to review the code associated with this book
it should be clearly stated that it is supplied as an extra--almost
as an afterthought. This may be a wise move.
Although not as bad as other (even commercial) implementations,
the code provided lacks proper OO structure and is typical of C++
written in a C style.
Style criticisms include:
In a generous sense the code is free and the author doesn't
claim any expertise in software engineering. It works in a limited
sense but would be difficult to extend and/or reuse. It's fine for
demonstration purposes in a stand-alone manner and for use with the
book concerned.
If you're serious about nets you'll end up rewriting the whole
lot (or getting something better).
Blum,
Adam (1992), Neural Networks in C++, NY: Wiley.
Welstead, Stephen T. (1994), Neural Network and Fuzzy Logic
Applications in C/C++, NY: Wiley.
Both Blum and Welstead contribute to the dangerous myth that any
idiot can use a neural net by dumping in whatever data are handy
and letting it train for a few days. They both have little or no
discussion of generalization, validation, and overfitting. Neither
provides any valid advice on choosing the number of hidden nodes.
If you have ever wondered where these stupid "rules of thumb" that
pop up frequently come from, here's a source for one of them:
Blum offers some profound advice on choosing inputs:
My comments apply only to the text of the above books. I have not
examined or attempted to compile the code.
Swingler,
K. (1996), Applying Neural Networks: A Practical Guide,
London: Academic Press.
This book has lots of good advice liberally sprinkled with errors,
incorrect formulas, some bad advice, and some very serious mistakes.
Experts will learn nothing, while beginners will be unable to separate
the useful information from the dangerous. For example, there is a
chapter on "Data encoding and re-coding" that would be very useful to
beginners if it were accurate, but the formula for the standard
deviation is wrong, and the description of the softmax function is of
something entirely different than softmax (see
What is a softmax activation function?).
Even more dangerous is the statement on p. 28 that "Any pair of
variables with high covariance are dependent, and one may be chosen to
be discarded." Although high correlations can be used to identify
redundant inputs, it is incorrect to use high covariances for this
purpose, since a covariance can be high simply because one of the inputs
has a high standard deviation.
The most ludicrous thing I've found in the book is the claim that
Hecht-Neilsen used Kolmogorov's theorem to show that "you will never
require more than twice the number of hidden units as you have inputs"
(p. 53) in an MLP with one hidden layer. Actually, Hecht-Neilsen, says
"the direct usefulness of this result is doubtful, because no
constructive method for developing the [output activation] functions is
known." Then Swingler implies that V. Kurkova (1991, "Kolmogorov's
theorem is relevant," Neural Computation, 3, 617-622) confirmed this
alleged upper bound on the number of hidden units, saying that, "Kurkova
was able to restate Kolmogorov's theorem in terms of a set of sigmoidal
functions." If Kolmogorov's theorem, or Hecht-Nielsen's adaptation of
it, could be restated in terms of known sigmoid activation functions in
the (single) hidden and output layers, then Swingler's alleged upper
bound would be correct, but in fact no such restatement of Kolmogorov's
theorem is possible, and Kurkova did not claim to prove any such
restatement. Swingler omits the crucial details that Kurkova used two
hidden layers, staircase-like activation functions (not ordinary
sigmoidal functions such as the logistic) in the first hidden layer, and
a potentially large number of units in the second hidden layer. Kurkova
later estimated the number of units required for uniform approximation
within an error epsilon as nm(m+1) in the first hidden
layer and m^2(m+1)^n in the second hidden layer, where
n is the number of inputs and m "depends on
epsilon/||f|| as well as on the rate with which f
increases distances." In other words, Kurkova says nothing to support
Swinglers advice (repeated on p. 55), "Never choose h to be more than
twice the number of input units." Furthermore, constructing a counter
example to Swingler's advice is trivial: use one input and one output,
where the output is the sine of the input, and the domain of the input
extends over many cycles of the sine wave; it is obvious that many more
than two hidden units are required.
For some sound information on choosing the number of hidden units, see
How many hidden units should I use?
Choosing the number of hidden units is one important aspect of getting
good generalization, which is the most crucial issue in neural network
training. There are many other considerations involved in getting good
generalization, and Swingler makes several more mistakes in this area:
Swingler addresses many important practical issues, and often provides
good practical advice. But the peculiar combination of much
good advice with some extremely bad advice, a few examples of which are
provided above, could easily seduce a beginner into thinking that
the book as a whole is reliable. It is this danger that earns the book
a place in "The Worst" list.
Dewdney,
A.K. (1997), Yes, We Have No Neutrons: An Eye-Opening Tour
through the Twists and Turns of Bad Science,
NY: Wiley.
You can post, read, and reply messages on the Web. Or you can
choose to receive messages as individual emails, daily summaries,
daily full-text digest, or read them on the Web only.
To subscribe, send an empty message to
machine-learning-subscribe@egroups.com
Archives are at http://www.egroups.com/group/machine-learning
Benchmark studies require some familiarity with the statistical
design and analysis of experiments. There are many textbooks on this
subject, of which Cohen (1995) will probably be of particular interest
to researchers in neural nets and machine learning (see also the
review of Cohen's book by Ron Kohavi in the International Journal of
Neural Systems, which can be found on-line at
http://robotics.stanford.edu/users/ronnyk/ronnyk-bib.html).
Reference:
Cohen, P.R. (1995), Empirical Methods for Artificial
Intelligence, Cambridge, MA: The MIT Press.
The system requirements for all databases are a 5.25" CD-ROM drive
with software to read ISO-9660 format.
Contact: Darrin L. Dimmick;
dld@magi.ncsl.nist.gov; (301)975-4147
The prices of the databases are between US$ 250 and 1895
If you wish to order a database, please contact:
Standard Reference Data;
National Institute of Standards and Technology;
221/A323;
Gaithersburg, MD 20899;
Phone: (301)975-2208; FAX: (301)926-0416
Samples of the data can be found by ftp on sequoyah.ncsl.nist.gov
in directory /pub/data
A more complete description of the available databases can be obtained
from the same host as
/pub/databases/catalog.txt
There is also a CEDAR CDROM-2, a database of machine-printed
Japanese character images.
Some of the datasets were used in a prediction contest and are described
in detail in the book "Time series prediction: Forecasting the future
and understanding the past", edited by Weigend/Gershenfield, Proceedings
Volume XV in the Santa Fe Institute Studies in the Sciences of Complexity
series of Addison Wesley (1994).
For further information, see
ftp://src.doc.ic.ac.uk/pub/packages/faces/README
Do NOT do a directory listing in the top directory of the face archive,
as it contains over 2500 entries!
According to the archive administrator, Barbara L. Dijker
(barb.dijker@labyrinth.com), there is no restriction to use them.
However, the image files are stored in separate directories
corresponding to the Internet site to which the person represented in
the image belongs, with each directory containing a small number of
images (two in the average). This makes it difficult to retrieve by ftp
even a small part of the database, as you have to get each one
individually.
Please consider that UUNET has graciously provided the ftp server for
the FaceSaver archive and may discontinue that service if it becomes a
burden. This means that people should not download more than maybe 10
faces at a time from uunet.
A last remark: each file represents a different person (except for
isolated cases). This makes the database quite unsuitable for training
neural networks, since for proper generalisation several instances of
the same subject are required. However, it is still useful for use as
testing set on a trained network.
CityU Image Processing Lab:
http://www.image.cityu.edu.hk/images/database.html
Center for Image Processing Research:
http://cipr.rpi.edu/
Computer Vision Test Images:
http://www.cs.cmu.edu:80/afs/cs/project/cil/ftp/html/v-images.html
Lenna 97: A Complete Story of Lenna:
http://www.image.cityu.edu.hk/images/lenna/Lenna97.html
This book has relatively little material explicitly about neural nets,
but what it has is very interesting and much of it is not found in other
texts. The emphasis is on statistical proofs of universal consistency
for a wide variety of methods, including histograms, (k) nearest neighbors, kernels
(PNN), trees, generalized linear discriminants, MLPs, and RBF networks.
There is also considerable material on validation and cross-validation.
The authors say, "We did not scar the pages with backbreaking
simulations or quick-and-dirty engineering solutions" (p. 7).
The formula-to-text ratio is high, but the writing is quite
clear, and anyone who has had a year or two of mathematical statistics
should be able to follow the exposition.
The best books on image and signal processing with NNs
Masters, T. (1994), Signal and Image Processing with Neural
Networks: A C++ Sourcebook, NY: Wiley.
Book Webpage (Publisher): http://www.wiley.com/
Additional Information: One has to search.
Book Webpage (Publisher): http://www.wiley.com/
Additional Information: One has to search.
Comments from readers of comp.ai.neural-nets:"Partly a textbook and
partly a research monograph; introduces the basic concepts, techniques,
and models related to neural networks and optimization, excluding
rigorous mathematical details. Accessible to a wide readership with a
differential calculus background. The main coverage of the book is on
recurrent neural networks with continuous state variables. The book
title would be more appropriate without mentioning signal processing.
Well edited, good illustrations."The best book on time-series forecasting with NNs
Weigend, A.S. and Gershenfeld, N.A., eds. (1994) Time Series Prediction:
Forecasting the Future and Understanding the Past, Reading, MA:
Addison-Wesley.
Book Webpage (Publisher):
http://www2.awl.com/gb/abp/sfi/complexity.html
The best books on reinforcement learning
Elementary:
Sutton, R.S., and Barto, A.G. (1998), Reinforcement Learning:
An Introduction, The MIT Press, ISBN: 0-262193-98-1.
Author's Webpage: http://envy.cs.umass.edu/~rich/sutton.html
and http://www-anw.cs.umass.edu/People/barto/barto.html
Book Webpage (Publisher):http://mitpress.mit.edu/book-home.tcl?isbn=0262193981
Additional Information:
http://www-anw.cs.umass.edu/~rich/book/the-book.html
Bertsekas, D. P. and Tsitsiklis, J. N. (1996), Neuro-Dynamic
Programming, Belmont, MA: Athena Scientific, ISBN 1-886529-10-8.
Author's Webpage: http://www.mit.edu:8001/people/dimitrib/home.html
and http://web.mit.edu/jnt/www/home.html
Book Webpage (Publisher):http://world.std.com/~athenasc/ndpbook.html
The best books on neurofuzzy systems
Brown, M., and Harris, C. (1994), Neurofuzzy Adaptive
Modelling and Control, NY: Prentice Hall.
Author's Webpage:
http://www.isis.ecs.soton.ac.uk/people/m_brown.html
and http://www.ecs.soton.ac.uk/~cjh/
Book Webpage (Publisher): http://www.prenhall.com/books/esm_0131344536.html
Additional Information: Additional page at:
http://www.isis.ecs.soton.ac.uk/publications/neural/mqbcjh94e.html and an
abstract can be found at:
http://www.isis.ecs.soton.ac.uk/publications/neural/mqb93.html
Brown and Harris rely on the fundamental insight that that a
fuzzy system is a nonlinear mapping from an input space to an output
space that can be parameterized in various ways and therefore can be
adapted to data using the usual neural training methods (see "What is backprop?") or conventional numerical
optimization algorithms (see "What are
conjugate gradients, Levenberg-Marquardt, etc.?"). Their approach
makes clear the intimate connections between fuzzy systems, neural
networks, and statistical methods such as B-spline regression.
Kosko's new book is a big improvement over his older neurofuzzy book
and makes an excellent sequel to Brown and Harris (1994).
The best comparison of NNs with other classification methods
Michie, D., Spiegelhalter, D.J. and Taylor, C.C. (1994), Machine
Learning, Neural and Statistical Classification, Ellis Horwood.
Author's Webpage: Donald Michie: http://www.aiai.ed.ac.uk/~dm/dm.html
Additional Information: This book is out of print but available online at
http://www.amsta.leeds.ac.uk/~charles/statlog/
Books for the Beginner
Aleksander, I. and Morton, H. (1990). An Introduction to Neural Computing.
Chapman and Hall. (ISBN 0-412-37780-2).
Book Webpage (Publisher): http://www.chaphall.com/
Additional Information: Seems to be out of print.
Comments from readers of comp.ai.neural-nets:: "This book seems to be
intended for the first year of university education."
Comments from readers of comp.ai.neural-nets: "It's clearly written.
Lots of hints as to how to get the adaptive models covered to work (not
always well explained in the original sources). Consistent mathematical
terminology. Covers perceptrons, error-backpropagation, Kohonen
self-org model, Hopfield type models, ART, and associative memories."
Book Webpage (Publisher): http://mitpress.mit.edu/book-home.tcl?isbn=0262531135
The authors try to translate mathematical formulas into English. The
results are likely to disturb people who appreciate either mathematics
or English. Have the authors never heard that "a picture is worth a
thousand words"? What few diagrams they have (such as the one on p. 74)
tend to be confusing. Their jargon is peculiar even by NN standards;
for example, they refer to target values as "mentor inputs" (p. 66).
The authors do not understand elementary properties of error functions
and optimization algorithms. For example, in their discussion of the
delta rule, the authors seem oblivious to the differences between batch
and on-line training, and they attribute magical properties to the
algorithm (p. 71):
[The on-line delta] rule always takes the most efficient route from the
current position of the weight vector to the "ideal" position, based
on the current input pattern. The delta rule not only minimizes the
mean squared error, it does so in the most efficient fashion
possible--quite an achievement for such a simple rule.
While the authors realize that backpropagation networks can suffer
from local minima, they mistakenly think that counterpropagation has
some kind of global optimization ability (p. 202):
Unlike the backpropagation network, a counterpropagation network cannot
be fooled into finding a local minimum solution. This means that the
network is guaranteed to find the correct response (or the nearest
stored response) to an input, no matter what.
But even though they acknowledge the problem of local minima, the
authors are ignorant of the importance of initial weight values (p. 186):
To teach our imaginary network something using backpropagation, we
must start by setting all the adaptive weights on all the neurodes
in it to random values. It won't matter what those values are, as
long as they are not all the same and not equal to 1.
Like most introductory books, this one
neglects the difficulties of getting good generalization--the authors
simply declare (p. 8) that "A neural network is able to generalize"!
Book Webpage (Publisher): http://www.prenhall.com/
Additional Information: Seems to be out of print.
Shallow, sometimes confused, especially with regard to Kohonen networks.
Comments from readers of comp.ai.neural-nets: "Like Wasserman's book, Dayhoff's book is also very easy to
understand".
Additional Information: Sourcecode available under:
ftp://ftp.mathsource.com/pub/Publications/BookSupplements/Freeman-1993
Helps the reader make his own NNs. The mathematica code for the programs in the book
is also available through the internet: Send mail to
MathSource@wri.com or try
http://www.wri.com/ on the World Wide
Web.
Book Webpage (Publisher): http://www.awl.com/
Additional Information: Seems to be out of print.
A good book for beginning programmers who want to learn how to write
NN programs while avoiding any understanding of what NNs do or why
they do it.
Book Webpage (Publisher): http://www.wiley.com/
Additional Information: One has to search.
Franco Insana comments:
* Decent book for the neural net beginner
* Very little devoted to statistical framework, although there
is some formulation of backprop theory
* Some food for thought
* Nothing here for those with any neural net experience
Book Webpage (Publisher): http://www.awl.com/
Additional Information: Seems to be out of print.
Comments from readers of comp.ai.neural-nets: "A good book", "comprises
a nice historical overview and a chapter about NN hardware. Well
structured prose. Makes important concepts clear."
Book Webpage (Publisher):
http://mitpress.mit.edu/book-home.tcl?isbn=026263113X (IBM version) and
http://mitpress.mit.edu/book-home.tcl?isbn=0262631296 (Macintosh)
Comments from readers of comp.ai.neural-nets: "Written in a tutorial
style, and includes 2 diskettes of NN simulation programs that can be
compiled on MS-DOS or Unix (and they do too !)"; "The programs are
pretty reasonable as an introduction to some of the things that NNs can
do."; "There are *two* editions of this book. One comes with disks for
the IBM PC, the other comes with disks for the Macintosh".
Book Webpage (Publisher):
http://cseng.aw.com/bookdetail.qry?ISBN=0-201-63378-7&ptype=1174
Lots of applications without technical details, lots of hype,
lots of goofs, no formulas.
Book Webpage (Publisher):
http://www.springer.de/catalog/html-files/deutsch/phys/3540602070.html
Comments from readers of comp.ai.neural-nets: "The book was developed
out of a course on neural-network models with computer demonstrations
that was taught by the authors to Physics students. The book comes
together with a PC-diskette. The book is divided into three parts: (1)
Models of Neural Networks; describing several architectures and learing
rules, including the mathematics. (2) Statistical Physiscs of Neural
Networks; "hard-core" physics section developing formal theories of
stochastic neural networks. (3) Computer Codes; explanation about the
demonstration programs. First part gives a nice introduction into
neural networks together with the formulas. Together with the
demonstration programs a 'feel' for neural networks can be developed."
Comments from readers of comp.ai.neural-nets: "Short user-friendly
introduction to the area, with a non-technical flavour. Apparently
accompanies a software package, but I haven't seen that yet".
Covers a wider variety of networks than Masters (1993), Practical
Neural Network Recipes in C++,, but lacks Masters's
insight into practical issues of using NNs.
Comments from readers of comp.ai.neural-nets: "Wasserman flatly
enumerates some common architectures from an engineer's perspective
('how it works') without ever addressing the underlying fundamentals
('why it works') - important basic concepts such as clustering,
principal components or gradient descent are not treated. It's also
full of errors, and unhelpful diagrams drawn with what appears to be PCB
board layout software from the '70s. For anyone who wants to do active
research in the field I consider it quite inadequate"; "Okay, but too
shallow"; "Quite easy to understand"; "The best bedtime reading for
Neural Networks. I have given this book to numerous collegues who want
to know NN basics, but who never plan to implement anything. An
excellent book to give your manager."The Classics
Kohonen, T. (1984). Self-organization and Associative Memory. Springer-Verlag:
New York. (2nd Edition: 1988; 3rd edition: 1989).
Author's Webpage: http://www.cis.hut.fi/nnrc/teuvo.html
Book Webpage (Publisher): http://www.springer.de/
Additional Information: Book is out of print.
Comments from readers of comp.ai.neural-nets: "The section on Pattern
mathematics is excellent."
Author's Webpage:
http://www-med.stanford.edu/school/Neurosciences/faculty/rumelhart.html
Book Webpage (Publisher):
http://mitpress.mit.edu/book-home.tcl?isbn=0262631121
Comments from readers of comp.ai.neural-nets: "As a computer scientist I
found the two Rumelhart and McClelland books really heavy going and
definitely not the sort of thing to read if you are a beginner."; "It's
quite readable, and affordable (about $65 for both volumes)."; "THE
Connectionist bible".Introductory Journal Articles
Hinton, G. E. (1989). Connectionist learning procedures.
Artificial Intelligence, Vol. 40, pp. 185--234.
Author's Webpage:
http://www.cs.utoronto.ca/DCS/People/Faculty/hinton.html (official) and
http://www.cs.toronto.edu/~hinton (private)
Journal Webpage (Publisher): http://www.elsevier.nl/locate/artint
Comments from readers of comp.ai.neural-nets: "One of the better neural
networks overview papers, although the distinction between network
topology and learning algorithm is not always very clear. Could very
well be used as an introduction to neural networks."
Comments from readers of comp.ai.neural-nets:"A good article, while it
is for most people easy to find a copy of this journal."
Author's Webpage: http://www.cis.hut.fi/nnrc/teuvo.html
Journal Webpage (Publisher):
http://www.eeb.ele.tue.nl/neural/neural.html
Additional Information: Article not available there.
Comments from readers of comp.ai.neural-nets: "A general review".
Journal Webpage (Publisher): http://www.nature.com/
Additional Information: Article not available there.
Comments from readers of comp.ai.neural-nets: "Gives a very good potted
explanation of backprop NN's. It gives sufficient detail to write your
own NN simulation."Not-quite-so-introductory Literature
Anderson, J. A. and Rosenfeld, E. (Eds). (1988). Neurocomputing:
Foundations of Research. The MIT Press: Cambridge, MA.
Author's Webpage: http://www.cog.brown.edu/~anderson
Book Webpage (Publisher):
http://mitpress.mit.edu/book-home.tcl?isbn=0262510480
Comments from readers of comp.ai.neural-nets: "An expensive book, but
excellent for reference. It is a collection of reprints of most of the
major papers in the field."
Author's Webpage: http://www.cog.brown.edu/~anderson
Book Webpage (Publisher):
http://mitpress.mit.edu/book-home.tcl?isbn=0262510758
Comments from readers of comp.ai.neural-nets: "The sequel to their
well-known Neurocomputing book."
Comments from readers of comp.ai.neural-nets: "A very readable, well
written intermediate text on NNs Perspective is primarily one of pattern
recognition, estimation and signal processing. However, there are
well-written chapters on neurodynamics and VLSI implementation. Though
there is emphasis on formal mathematical models of NNs as universal
approximators, statistical estimators, etc., there are also examples of
NNs used in practical applications. The problem sets at the end of each
chapter nicely complement the material. In the bibliography are over
1000 references."
Book Webpage (Publisher): http://www.awl.com/
Comments from readers of comp.ai.neural-nets: "Not so bad (with a page
of erroneous formulas (if I remember well), and #hidden layers isn't
well described)."; "Khanna's intention in writing his book with math
analysis should be commended but he made several mistakes in the math
part".
Levine, D. S. (1990). Introduction to Neural and Cognitive Modeling.
Lawrence Erlbaum: Hillsdale, N.J.
Comments from readers of comp.ai.neural-nets: "Highly recommended".
Comments from readers of comp.ai.neural-nets: "Much acclaimed as an
overview of neural networks, but rather inaccurate on several points.
The categorization into binary and continuous- valued input neural
networks is rather arbitrary, and may work confusing for the
unexperienced reader. Not all networks discussed are of equal
importance."
Comments from readers of comp.ai.neural-nets: "They cover a broad area";
"Introductory with suggested applications implementation".
Book Webpage (Publisher): http://www.awl.com/
Comments from readers of comp.ai.neural-nets: "An excellent book that
ties together classical approaches to pattern recognition with Neural
Nets. Most other NN books do not even mention conventional
approaches."
Book Webpage (Publisher): http://www.wiley.com/
Additional Information: One has to search.
Franco Insana comments:
* Not for the beginner
* Excellent introductory material presented by editor in first 5
chapters, which could be a valuable reference source for any
practitioner
* Very thought-provoking
* Mostly backprop-related
* Most contributors lay good statistical foundation
* Overall, a wealth of information and ideas, but the reader has to
sift through it all to come away with anything useful
Comments from readers of comp.ai.neural-nets: "Contains a very useful 37
page bibliography. A large number of paradigms are presented. On the
negative side the book is very shallow. Best used as a complement to
other books".
Comments from readers of comp.ai.neural-nets: "Several neural network
topics are discussed e.g. Probalistic Neural Networks, Backpropagation
and beyond, neural control, Radial Basis Function Networks, Neural
Engineering. Furthermore, several subjects related to neural networks
are mentioned e.g. genetic algorithms, fuzzy logic, chaos. Just the
functionality of these subjects is described; enough to get you started.
Lots of references are given to more elaborate descriptions. Easy to
read, no extensive mathematical background necessary."
Comments from readers of comp.ai.neural-nets: "Gives the AI point of
view".
Comments from readers of comp.ai.neural-nets: "Covers quite a broad
range of topics (collection of articles/papers )."; "Provides a
primer-like introduction and overview for a broad audience, and employs
a strong interdisciplinary emphasis".
Comments from readers of comp.ai.neural-nets: "Cohesive and
comprehensive book on neural nets; as an engineering-oriented
introduction, but also as a research foundation. Thorough exposition of
fundamentals, theory and applications. Training and recall algorithms
appear in boxes showing steps of algorithms, thus making programming of
learning paradigms easy. Many illustrations and intuitive examples.
Winner among NN textbooks at a senior UG/first year graduate level-[175
problems]." Contents: Intro, Fundamentals of Learning, Single-Layer &
Multilayer Perceptron NN, Assoc. Memories, Self-organizing and Matching
Nets, Applications, Implementations, Appendix) Books with Source Code (C, C++)
Blum,
Adam (1992), Neural Networks in C++, Wiley.
Review by Ian Cresswell. (For a review of the text, see
"The Worst" below.)
... garble(float noise) ...
Swingler,
K. (1996), Applying Neural Networks: A Practical Guide,
London: Academic Press.
Review by Ian Cresswell. (For a review of the text, see
"The Worst" below.)
This code is without doubt written by a rushed C programmer.
Whilst it would require a C++ compiler to be successfully used, it
lacks the tight (optimised) nature of good C and the high level of
abstraction of good C++. The Worst
How not to use neural nets in any programming language
(For a review of Blum's source code, see
"Books with Source Code" above.)
"A rule of thumb is for the size of this [hidden] layer to be
somewhere between the input layer size ... and the output layer
size ..." Blum, p. 60.
(John Lazzaro tells me he recently "reviewed a paper that cited this rule of
thumb--and referenced this book! Needless to say, the final version of that paper
didn't include the reference!")
"The next step is to pick as many input factors as possible that
might be related to [the target]."
Blum also shows a deep understanding of statistics:
"A statistical model is simply a more indirect way of learning
correlations. With a neural net approach, we model the problem
directly." p. 8.
Blum at least mentions some important issues, however simplistic his
advice may be. Welstead just ignores them. What Welstead gives you is
code--vast amounts of code. I have no idea how anyone could write that
much code for a simple feedforward NN. Welstead's approach to
validation, in his chapter on financial forecasting, is to reserve
two cases for the validation set! An impractical guide to neural nets
(For a review of the source code, see
"Books with Source Code" above.)
While Swingler has some knowldege of statistics, his expertise is not
sufficient for him to detect that certain articles on neural nets are
statistically nonsense. For example, on pp. 139-140 he uncritically
reports a method that allegedly obtains error bars by doing a simple
linear regression on the target vs. output scores. To a trained
statistician, this method is obviously wrong (and, as usual in this
book, the formula for variance given for this method on p. 150 is
wrong). On p. 110, Swingler reports an article that attempts to apply
bootstrapping to neural nets, but this article is also obviously wrong
to anyone familiar with bootstrapping. While Swingler cannot be blamed
entirely for accepting these articles at face value, such misinformation
provides yet more hazards for beginners. Bad science writing
This book, allegedly an expose of bad science, contains only one chapter
of 19 pages on "the neural net debacle" (p. 97). Yet this chapter is so
egregiously misleading that the book has earned a place on "The Worst"
list. A detailed criticism of this chapter, along with some other
sections of the book, can be found at
ftp://ftp.sas.com/pub/neural/badscience.html.
Other chapters of the book are reviewed in the November, 1997, issue of
Scientific American.
------------------------------------------------------------------------
Subject: Journals and magazines about
Neural Networks?
[to be added: comments on speed of reviewing and publishing,
whether they accept TeX format or ASCII by e-mail, etc.]
A. Dedicated Neural Network Journals:
Title: Neural Networks
Publish: Pergamon Press
Address: Pergamon Journals Inc., Fairview Park, Elmsford,
New York 10523, USA and Pergamon Journals Ltd.
Headington Hill Hall, Oxford OX3, 0BW, England
Freq.: 10 issues/year (vol. 1 in 1988)
Cost/Yr: Free with INNS or JNNS or ENNS membership ($45?),
Individual $65, Institution $175
ISSN #: 0893-6080
WWW: http://www.elsevier.nl/locate/inca/841
Remark: Official Journal of International Neural Network Society (INNS),
European Neural Network Society (ENNS) and Japanese Neural
Network Society (JNNS).
Contains Original Contributions, Invited Review Articles, Letters
to Editor, Book Reviews, Editorials, Announcements, Software Surveys.
Title: Neural Computation
Publish: MIT Press
Address: MIT Press Journals, 55 Hayward Street Cambridge,
MA 02142-9949, USA, Phone: (617) 253-2889
Freq.: Quarterly (vol. 1 in 1989)
Cost/Yr: Individual $45, Institution $90, Students $35; Add $9 Outside USA
ISSN #: 0899-7667
URL: http://mitpress.mit.edu/journals-legacy.tcl
Remark: Combination of Reviews (10,000 words), Views (4,000 words)
and Letters (2,000 words). I have found this journal to be of
outstanding quality.
(Note: Remarks supplied by Mike Plonski "plonski@aero.org")
Title: IEEE Transactions on Neural Networks
Publish: Institute of Electrical and Electronics Engineers (IEEE)
Address: IEEE Service Cemter, 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ,
08855-1331 USA. Tel: (201) 981-0060
Cost/Yr: $10 for Members belonging to participating IEEE societies
Freq.: Quarterly (vol. 1 in March 1990)
URL: http://www.ieee.org/nnc/pubs/transactions.html
Remark: Devoted to the science and technology of neural networks
which disclose significant technical knowledge, exploratory
developments and applications of neural networks from biology to
software to hardware. Emphasis is on artificial neural networks.
Specific aspects include self organizing systems, neurobiological
connections, network dynamics and architecture, speech recognition,
electronic and photonic implementation, robotics and controls.
Includes Letters concerning new research results.
(Note: Remarks are from journal announcement)
Title: IEEE Transactions on Evolutionary Computation
Publish: Institute of Electrical and Electronics Engineers (IEEE)
Address: IEEE Service Cemter, 445 Hoes Lane, P.O. Box 1331, Piscataway, NJ,
08855-1331 USA. Tel: (201) 981-0060
Cost/Yr: $10 for Members belonging to participating IEEE societies
Freq.: Quarterly (vol. 1 in May 1997)
URL: http://engine.ieee.org/nnc/pubs/transactions.html
Remark: The IEEE Transactions on Evolutionary Computation will publish archival
journal quality original papers in evolutionary computation and related
areas, with particular emphasis on the practical application of the
techniques to solving real problems in industry, medicine, and other
disciplines. Specific techniques include but are not limited to
evolution strategies, evolutionary programming, genetic algorithms, and
associated methods of genetic programming and classifier systems. Papers
emphasizing mathematical results should ideally seek to put these results
in the context of algorithm design, however purely theoretical papers will
be considered. Other papers in the areas of cultural algorithms, artificial
life, molecular computing, evolvable hardware, and the use of simulated
evolution to gain a better understanding of naturally evolved systems are
also encouraged.
(Note: Remarks are from journal CFP)
Title: International Journal of Neural Systems
Publish: World Scientific Publishing
Address: USA: World Scientific Publishing Co., 1060 Main Street, River Edge,
NJ 07666. Tel: (201) 487 9655; Europe: World Scientific Publishing
Co. Ltd., 57 Shelton Street, London WC2H 9HE, England.
Tel: (0171) 836 0888; Asia: World Scientific Publishing Co. Pte. Ltd.,
1022 Hougang Avenue 1 #05-3520, Singapore 1953, Rep. of Singapore
Tel: 382 5663.
Freq.: Quarterly (Vol. 1 in 1990)
Cost/Yr: Individual $122, Institution $255 (plus $15-$25 for postage)
ISSN #: 0129-0657 (IJNS)
Remark: The International Journal of Neural Systems is a quarterly
journal which covers information processing in natural
and artificial neural systems. Contributions include research papers,
reviews, and Letters to the Editor - communications under 3,000
words in length, which are published within six months of receipt.
Other contributions are typically published within nine months.
The journal presents a fresh undogmatic attitude towards this
multidisciplinary field and aims to be a forum for novel ideas and
improved understanding of collective and cooperative phenomena with
computational capabilities.
Papers should be submitted to World Scientific's UK office. Once a
paper is accepted for publication, authors are invited to e-mail
the LaTeX source file of their paper in order to expedite publication.
Title: International Journal of Neurocomputing
Publish: Elsevier Science Publishers, Journal Dept.; PO Box 211;
1000 AE Amsterdam, The Netherlands
Freq.: Quarterly (vol. 1 in 1989)
WWW: http://www.elsevier.nl/locate/inca/505628
Title: Neural Processing Letters
Publish: Kluwer Academic publishers
Address: P.O. Box 322, 3300 AH Dordrecht, The Netherlands
Freq: 6 issues/year (vol. 1 in 1994)
Cost/Yr: Individuals $198, Institution $400 (including postage)
ISSN #: 1370-4621
URL: http://www.wkap.nl/journalhome.htm/1370-4621
Remark: The aim of the journal is to rapidly publish new ideas, original
developments and work in progress. Neural Processing Letters
covers all aspects of the Artificial Neural Networks field.
Publication delay is about 3 months.
Title: Neural Network News
Publish: AIWeek Inc.
Address: Neural Network News, 2555 Cumberland Parkway, Suite 299,
Atlanta, GA 30339 USA. Tel: (404) 434-2187
Freq.: Monthly (beginning September 1989)
Cost/Yr: USA and Canada $249, Elsewhere $299
Remark: Commercial Newsletter
Title: Network: Computation in Neural Systems
Publish: IOP Publishing Ltd
Address: Europe: IOP Publishing Ltd, Techno House, Redcliffe Way, Bristol
BS1 6NX, UK; IN USA: American Institute of Physics, Subscriber
Services 500 Sunnyside Blvd., Woodbury, NY 11797-2999
Freq.: Quarterly (1st issue 1990)
Cost/Yr: USA: $180, Europe: 110 pounds
Remark: Description: "a forum for integrating theoretical and experimental
findings across relevant interdisciplinary boundaries." Contents:
Submitted articles reviewed by two technical referees paper's
interdisciplinary format and accessability." Also Viewpoints and
Reviews commissioned by the editors, abstracts (with reviews) of
articles published in other journals, and book reviews.
Comment: While the price discourages me (my comments are based
upon a free sample copy), I think that the journal succeeds
very well. The highest density of interesting articles I
have found in any journal.
(Note: Remarks supplied by kehoe@csufres.CSUFresno.EDU)
Title: Connection Science: Journal of Neural Computing,
Artificial Intelligence and Cognitive Research
Publish: Carfax Publishing
Address: Europe: Carfax Publishing Company, PO Box 25, Abingdon, Oxfordshire
OX14 3UE, UK.
USA: Carfax Publishing Company, PO Box 2025, Dunnellon, Florida
34430-2025, USA
Australia: Carfax Publishing Company, Locked Bag 25, Deakin,
ACT 2600, Australia
Freq.: Quarterly (vol. 1 in 1989)
Cost/Yr: Personal rate:
48 pounds (EC) 66 pounds (outside EC) US$118 (USA and Canada)
Institutional rate:
176 pounds (EC) 198 pounds (outside EC) US$340 (USA and Canada)
Title: International Journal of Neural Networks
Publish: Learned Information
Freq.: Quarterly (vol. 1 in 1989)
Cost/Yr: 90 pounds
ISSN #: 0954-9889
Remark: The journal contains articles, a conference report (at least the
issue I have), news and a calendar.
(Note: remark provided by J.R.M. Smits "anjos@sci.kun.nl")
Title: Sixth Generation Systems (formerly Neurocomputers)
Publish: Gallifrey Publishing
Address: Gallifrey Publishing, PO Box 155, Vicksburg, Michigan, 49097, USA
Tel: (616) 649-3772, 649-3592 fax
Freq. Monthly (1st issue January, 1987)
ISSN #: 0893-1585
Editor: Derek F. Stubbs
Cost/Yr: $79 (USA, Canada), US$95 (elsewhere)
Remark: Runs eight to 16 pages monthly. In 1995 will go to floppy disc-based
publishing with databases +, "the equivalent to 50 pages per issue are
planned." Often focuses on specific topics: e.g., August, 1994 contains two
articles: "Economics, Times Series and the Market," and "Finite Particle
Analysis - [part] II." Stubbs also directs the company Advanced Forecasting
Technologies. (Remark by Ed Rosenfeld: ier@aol.com)
Title: JNNS Newsletter (Newsletter of the Japan Neural Network Society)
Publish: The Japan Neural Network Society
Freq.: Quarterly (vol. 1 in 1989)
Remark: (IN JAPANESE LANGUAGE) Official Newsletter of the Japan Neural
Network Society(JNNS)
(Note: remarks by Osamu Saito "saito@nttica.NTT.JP")
Title: Neural Networks Today
Remark: I found this title in a bulletin board of october last year.
It was a message of Tim Pattison, timpatt@augean.OZ
(Note: remark provided by J.R.M. Smits "anjos@sci.kun.nl")
Title: Computer Simulations in Brain Science
Title: Internation Journal of Neuroscience
Title: Neural Network Computation
Remark: Possibly the same as "Neural Computation"
Title: Neural Computing and Applications
Freq.: Quarterly
Publish: Springer Verlag
Cost/yr: 120 Pounds
Remark: Is the journal of the Neural Computing Applications Forum.
Publishes original research and other information
in the field of practical applications of neural computing.
B. NN Related Journals:
Title: Complex Systems
Publish: Complex Systems Publications
Address: Complex Systems Publications, Inc., P.O. Box 6149, Champaign,
IL 61821-8149, USA
Freq.: 6 times per year (1st volume is 1987)
ISSN #: 0891-2513
Cost/Yr: Individual $75, Institution $225
Remark: Journal COMPLEX SYSTEMS devotes to rapid publication of research
on science, mathematics, and engineering of systems with simple
components but complex overall behavior. Send mail to
"jcs@jaguar.ccsr.uiuc.edu" for additional info.
(Remark is from announcement on Net)
Title: Biological Cybernetics (Kybernetik)
Publish: Springer Verlag
Remark: Monthly (vol. 1 in 1961)
Title: Various IEEE Transactions and Magazines
Publish: IEEE
Remark: Primarily see IEEE Trans. on System, Man and Cybernetics;
Various Special Issues: April 1990 IEEE Control Systems
Magazine.; May 1989 IEEE Trans. Circuits and Systems.;
July 1988 IEEE Trans. Acoust. Speech Signal Process.
Title: The Journal of Experimental and Theoretical Artificial Intelligence
Publish: Taylor & Francis, Ltd.
Address: London, New York, Philadelphia
Freq.: ? (1st issue Jan 1989)
Remark: For submission information, please contact either of the editors:
Eric Dietrich Chris Fields
PACSS - Department of Philosophy Box 30001/3CRL
SUNY Binghamton New Mexico State University
Binghamton, NY 13901 Las Cruces, NM 88003-0001
dietrich@bingvaxu.cc.binghamton.edu cfields@nmsu.edu
Title: The Behavioral and Brain Sciences
Publish: Cambridge University Press
Remark: (Expensive as hell, I'm sure.)
This is a delightful journal that encourages discussion on a
variety of controversial topics. I have especially enjoyed
reading some papers in there by Dana Ballard and Stephen
Grossberg (separate papers, not collaborations) a few years
back. They have a really neat concept: they get a paper,
then invite a number of noted scientists in the field to
praise it or trash it. They print these commentaries, and
give the author(s) a chance to make a rebuttal or
concurrence. Sometimes, as I'm sure you can imagine, things
get pretty lively. I'm reasonably sure they are still at
it--I think I saw them make a call for reviewers a few
months ago. Their reviewers are called something like
Behavioral and Brain Associates, and I believe they have to
be nominated by current associates, and should be fairly
well established in the field. That's probably more than I
really know about it but maybe if you post it someone who
knows more about it will correct any errors I have made.
The main thing is that I liked the articles I read. (Note:
remarks by Don Wunsch
C. Journals loosely related to NNs:
Title: JOURNAL OF COMPLEXITY
Remark: (Must rank alongside Wolfram's Complex Systems)
Title: IEEE ASSP Magazine
Remark: (April 1987 had the Lippmann intro. which everyone likes to cite)
Title: ARTIFICIAL INTELLIGENCE
Remark: (Vol 40, September 1989 had the survey paper by Hinton)
Title: COGNITIVE SCIENCE
Remark: (the Boltzmann machine paper by Ackley et al appeared here
in Vol 9, 1983)
Title: COGNITION
Remark: (Vol 28, March 1988 contained the Fodor and Pylyshyn
critique of connectionism)
Title: COGNITIVE PSYCHOLOGY
Remark: (no comment!)
Title: JOURNAL OF MATHEMATICAL PSYCHOLOGY
Remark: (several good book reviews)
------------------------------------------------------------------------
Subject: Conferences and Workshops on
Neural Networks?
------------------------------------------------------------------------
Subject: Neural Network Associations?
You can find nice lists of NN societies in the WWW at
http://www.emsl.pnl.gov:2080/proj/neuron/neural/societies.html and at
http://www.ieee.org:80/nnc/research/othernnsoc.html. International Neural Network Society (INNS).
INNS membership includes subscription to "Neural Networks",
the official journal of the society.
Membership is $55 for non-students and $45 for students per year.
Address: INNS Membership, P.O. Box 491166, Ft. Washington, MD 20749.
International Student Society for Neural Networks (ISSNNets).
Membership is $5 per year.
Address: ISSNNet, Inc., P.O. Box 15661, Boston, MA 02215 USA
Women In Neural Network Research and technology (WINNERS).
Address: WINNERS, c/o Judith Dayhoff, 11141 Georgia Ave., Suite 206,
Wheaton, MD 20902. Phone: 301-933-9000.
European Neural Network Society (ENNS)
ENNS membership includes subscription to "Neural Networks",
the official journal of the society.
Membership is currently (1994) 50 UK pounds (35 UK pounds for
students) per year.
Address: ENNS Membership, Centre for Neural Networks,
King's College London, Strand, London WC2R 2LS, United Kingdom.
Japanese Neural Network Society (JNNS)
Address: Japanese Neural Network Society;
Department of Engineering, Tamagawa University;
6-1-1, Tamagawa Gakuen, Machida City, Tokyo;
194 JAPAN;
Phone: +81 427 28 3457, Fax: +81 427 28 3597
Association des Connexionnistes en THese (ACTH)
(the French Student Association for Neural Networks);
Membership is 100 FF per year;
Activities: newsletter, conference (every year), list of members,
electronic forum; Journal 'Valgo' (ISSN 1243-4825);
WWW page:
http://www.supelec-rennes.fr/acth/welcome.html ;
Contact: acth@loria.fr
Neurosciences et Sciences de l'Ingenieur (NSI)
Biology & Computer Science
Activity : conference (every year)
Address : NSI - TIRF / INPG
46 avenue Felix Viallet
38031 Grenoble Cedex
FRANCE
IEEE Neural Networks Council
Web page at http://www.ieee.org/nnc
SNN (Foundation for Neural Networks)
The Foundation for Neural Networks (SNN)
is a university based non-profit organization that
stimulates basic and applied research on neural networks in the Netherlands.
Every year SNN orgines a symposium on Neural Networks.
See http://www.mbfys.kun.nl/SNN/.
------------------------------------------------------------------------
Subject: On-line and machine-readable information about NNs?
See also "Other NN links?"
Neuron Digest
Internet Mailing List. From the welcome blurb:
"Neuron-Digest is a list (in digest form) dealing with all aspects
of neural networks (and any type of network or neuromorphic system)"
To subscribe, send email to neuron-request@psych.upenn.edu.
The ftp archives (including back issues) are available from
psych.upenn.edu in pub/Neuron-Digest or by sending email to
"archive-server@psych.upenn.edu".
comp.ai.neural-net readers also find the messages in that newsgroup
in the form of digests. Usenet groups comp.ai.neural-nets (Oha!)
and comp.theory.self-org-sys.
There is a periodic posting on comp.ai.neural-nets sent by
srctran@world.std.com (Gregory Aharonian) about Neural Network
patents. USENET newsgroup comp.org.issnnet
Forum for discussion of academic/student-related issues in NNs, as
well as information on ISSNNet (see question
"associations") and its activities. Central Neural System Electronic Bulletin Board
URL: ftp://inia.cls.org/pub/CNS/bbs/
Supported by: Wesley R. Elsberry
4160 Pirates' Beach,
Galveston, TX 77554
Email: welsberr@inia.cls.org
Alternative URL: http://128.2.209.79/afs/cs.cmu.edu/project/ai-repository/ai/areas/neural/cns/0.html
Many MS-DOS PD and shareware simulations, source code, benchmarks,
demonstration packages, information files; some Unix, Macintosh,
Amiga related files. Also available are files on AI, AI Expert
listings 1986-1991, fuzzy logic, genetic algorithms, artificial
life, evolutionary biology, and many Project Gutenberg and Wiretap
etexts. AI CD-ROM
Network Cybernetics Corporation produces the "AI CD-ROM". It is
an ISO-9660 format CD-ROM and contains a large assortment of
software related to artificial intelligence, artificial life, virtual
reality, and other topics. Programs for OS/2, MS-DOS, Macintosh, UNIX,
and other operating systems are included. Research papers, tutorials,
and other text files are included in ASCII, RTF, and other universal
formats. The files have been collected from AI bulletin boards,
Internet archive sites, University computer deptartments, and
other government and civilian AI research organizations. Network
Cybernetics Corporation intends to release annual revisions to the
AI CD-ROM to keep it up to date with current developments in the field.
The AI CD-ROM includes collections of files that address many
specific AI/AL topics including Neural Networks (Source code and
executables for many different platforms including Unix, DOS, and
Macintosh. ANN development tools, example networks, sample data,
tutorials. A complete collection of Neural Digest is included as well.)
The AI CD-ROM may be ordered directly by check, money order, bank
draft, or credit card from:
Network Cybernetics Corporation;
4201 Wingren Road Suite 202;
Irving, TX 75062-2763;
Tel 214/650-2002;
Fax 214/650-1929;
The cost is $129 per disc + shipping ($5/disc domestic or $10/disc foreign)
(See the comp.ai FAQ for further details) Machine Learning mailing list
The Machine Learning mailing list is an unmoderated mailing list
intended for people in Computer Sciences, Statistics,
Mathematics, and other areas or disciplines with interests in
Machine Learning. Researchers, practitioners, and users of
Machine Learning in academia, industry, and government are
encouraged to join the list to discuss and exchange ideas
regarding any aspect of Machine Learning, e.g., various learning
algorithms, data pre-processing, variable selection mechanism,
instance selection, and applications to real-world problems. Machine Learning Papers
http://gubbio.cs.berkeley.edu/mlpapers/
------------------------------------------------------------------------
Subject: How to benchmark learning methods?
The NN benchmarking resources page at
http://wwwipd.ira.uka.de/~prechelt/NIPS_bench.html
was created after a NIPS 1995 workshop on NN benchmarking. The page
contains pointers to various papers on proper benchmarking methodology
and to various sources of datasets.
------------------------------------------------------------------------
Subject: Databases for experimentation with NNs?
UCI machine learning database
A large collection of data sets
accessible via anonymous FTP at ftp.ics.uci.edu [128.195.1.1]
in directory
/pub/machine-learning-databases" or via web browser at
http://www.ics.uci.edu/~mlearn/MLRepository.html UCI KDD Archive
The UC Irvine Knowledge Discovery in Databases (KDD) Archive at
http://kdd.ics.uci.edu/ is an
online repository of large datasets which encompasses a wide variety
of data types, analysis tasks, and application areas. The primary
role of this repository is to serve as a benchmark testbed to enable
researchers in knowledge discovery and data mining to scale existing
and future data analysis algorithms to very large and complex data
sets. This archive is supported by the Information and Data
Management Program at the National Science Foundation, and is
intended to expand the current UCI Machine Learning Database
Repository to datasets that are orders of magnitude larger and more
complex. The neural-bench Benchmark collection
Accessible WWW at
http://www.boltz.cs.cmu.edu/
or via anonymous FTP at
ftp://ftp.boltz.cs.cmu.edu/pub/neural-bench/.
In case of problems or if you want to donate data,
email contact is "neural-bench@cs.cmu.edu".
The data sets in this repository include the 'nettalk' data,
'two spirals', protein structure prediction, vowel recognition,
sonar signal classification, and a few others. Proben1
Proben1 is a collection of 12 learning problems consisting of real data.
The datafiles all share a single simple common format.
Along with the data comes a technical report describing a set
of rules and conventions for performing and reporting benchmark tests
and their results.
Accessible via anonymous FTP on ftp.cs.cmu.edu [128.2.206.173] as
/afs/cs/project/connect/bench/contrib/prechelt/proben1.tar.gz.
and also on ftp.ira.uka.de as
/pub/neuron/proben1.tar.gz.
The file is about 1.8 MB and unpacks into about 20 MB. Delve: Data for Evaluating Learning in Valid Experiments
Delve is a standardised, copyrighted environment designed to evaluate
the performance of learning methods. Delve makes it possible for users
to compare their learning methods with other methods on many datasets.
The Delve learning methods and evaluation procedures are well
documented, such that meaningful comparisons can be made. The data
collection includes not only isolated data sets, but "families" of data
sets in which properties of the data, such as number of inputs and
degree of nonlinearity or noise, are systematically varied. The Delve
web page is at
http://www.cs.toronto.edu/~delve/
NIST special databases of the National Institute Of Standards
And Technology:
Several large databases, each delivered on a CD-ROM. Here is a quick list.
Here are example descriptions of two of these databases:
NIST special database 2:
Structured Forms Reference Set (SFRS)
The NIST database of structured forms contains 5,590 full page images
of simulated tax forms completed using machine print. THERE IS NO REAL
TAX DATA IN THIS DATABASE. The structured forms used in this database
are 12 different forms from the 1988, IRS 1040 Package X. These
include Forms 1040, 2106, 2441, 4562, and 6251 together with Schedules
A, B, C, D, E, F and SE. Eight of these forms contain two pages or
form faces making a total of 20 form faces represented in the
database. Each image is stored in bi-level black and white raster
format. The images in this database appear to be real forms prepared
by individuals but the images have been automatically derived and
synthesized using a computer and contain no "real" tax data. The entry
field values on the forms have been automatically generated by a
computer in order to make the data available without the danger of
distributing privileged tax information. In addition to the images
the database includes 5,590 answer files, one for each image. Each
answer file contains an ASCII representation of the data found in the
entry fields on the corresponding image. Image format documentation
and example software are also provided. The uncompressed database
totals approximately 5.9 gigabytes of data.
NIST special database 3:
Binary Images of Handwritten Segmented Characters (HWSC)
Contains 313,389 isolated character images segmented from the
2,100 full-page images distributed with "NIST Special Database 1".
223,125 digits, 44,951 upper-case, and 45,313 lower-case character
images. Each character image has been centered in a separate
128 by 128 pixel region, error rate of the segmentation and
assigned classification is less than 0.1%.
The uncompressed database totals approximately 2.75 gigabytes of
image data and includes image format documentation and example
software. CEDAR CD-ROM 1: Database of Handwritten
Cities, States, ZIP Codes, Digits, and Alphabetic Characters
The Center Of Excellence for Document Analysis and Recognition (CEDAR)
State University of New York at Buffalo announces the availability of
CEDAR CDROM 1: USPS Office of Advanced Technology
The database contains handwritten words and ZIP Codes
in high resolution grayscale (300 ppi 8-bit) as well as
binary handwritten digits and alphabetic characters (300 ppi
1-bit). This database is intended to encourage research in
off-line handwriting recognition by providing access to
handwriting samples digitized from envelopes in a working
post office.
Specifications of the database include:
+ 300 ppi 8-bit grayscale handwritten words (cities,
states, ZIP Codes)
o 5632 city words
o 4938 state words
o 9454 ZIP Codes
+ 300 ppi binary handwritten characters and digits:
o 27,837 mixed alphas and numerics segmented
from address blocks
o 21,179 digits segmented from ZIP Codes
+ every image supplied with a manually determined
truth value
+ extracted from live mail in a working U.S. Post
Office
+ word images in the test set supplied with dic-
tionaries of postal words that simulate partial
recognition of the corresponding ZIP Code.
+ digit images included in test set that simulate
automatic ZIP Code segmentation. Results on these
data can be projected to overall ZIP Code recogni-
tion performance.
+ image format documentation and software included
System requirements are a 5.25" CD-ROM drive with software to read
ISO-9660 format.
For further information, see
http://www.cedar.buffalo.edu/Databases/CDROM1/
or send email to Ajay Shekhawat at <ajay@cedar.Buffalo.EDU> AI-CD-ROM (see question "Other sources of
information")
Time series archive
Various datasets of time series (to be used for prediction learning
problems) are available for anonymous ftp from ftp.santafe.edu [192.12.12.1]
in
/pub/Time-Series".
Problems are for example: fluctuations in a far-infrared laser;
Physiological data of patients with sleep apnea;
High frequency currency exchange rate data;
Intensity of a white dwarf star;
J.S. Bachs final (unfinished) fugue from "Die Kunst der Fuge" USENIX Faces
The USENIX faces archive is a public database, accessible by ftp, that
can be of use to people working in the fields of human face recognition,
classification and the like. It currently contains 5592 different faces
(taken at USENIX conferences) and is updated twice each year. The images
are mostly 96x128 greyscale frontal images and are stored in ascii files
in a way that makes it easy to convert them to any usual graphic format
(GIF, PCX, PBM etc.). Source code for viewers, filters, etc. is
provided. Each image file takes approximately 25K.
A solution, as Barbara proposed me, would be to compress the whole set
of images (in separate files of, say, 100 images) and maintain them as a
specific archive for research on face processing, similar to the ones
that already exist for fingerprints and others. The whole compressed
database would take some 30 megabytes of disk space. I encourage anyone
willing to host this database in his/her site, available for anonymous
ftp, to contact her for details (unfortunately I don't have the
resources to set up such a site). Linguistic Data Consortium
The Linguistic Data Consortium
(URL: http://www.ldc.upenn.edu/ldc/noframe.html)
is an open consortium of universities, companies and government research
laboratories. It creates, collects and distributes speech and text
databases, lexicons, and other resources for research and development
purposes. The University of Pennsylvania is the LDC's host institution.
The LDC catalog includes pronunciation lexicons, varied lexicons,
broadcast speech, microphone speech, mobile-radio speech,
telephone speech, broadcast text, conversation text, newswire text,
parallel text, and varied text, at widely varying fees.
Linguistic Data Consortium
University of Pennsylvania
3615 Market Street, Suite 200
Philadelphia, PA 19104-2608
Tel (215) 898-0464 Fax (215) 573-2175
Email: ldc@ldc.upenn.edu
Otago Speech Corpus
The Otago Speech Corpus contains speech samples in RIFF WAVE format
that can be downloaded from
http://divcom.otago.ac.nz/infosci/kel/software/RICBIS/hyspeech_main.html
Astronomical Time Series
Prepared by Paul L. Hertz (Naval Research Laboratory) & Eric D. Feigelson (Pennsyvania State University):
URL: http://xweb.nrl.navy.mil/www_hertz/timeseries/timeseries.html
Miscellaneous Images
The USC-SIPI Image Database:
http://sipi.usc.edu/services/database/Database.html StatLib
The StatLib repository at
http://lib.stat.cmu.edu/
at Carnegie Mellon University has a large collection of data sets,
many of which can be used with NNs.
------------------------------------------------------------------------
Next part is part 5 (of 7).
Previous part is part 3.